Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 3956 - tar is very slow when /etc/passwd is very big
Summary: tar is very slow when /etc/passwd is very big
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: tar
Version: 6.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: David Lawrence
QA Contact:
URL: http://www.engin.umich.edu/caen/syste...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 1999-07-09 04:11 UTC by wingc
Modified: 2008-05-01 15:37 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 1999-07-13 22:00:06 UTC


Attachments (Terms of Use)

Description wingc 1999-07-09 04:11:42 UTC
When /etc/passwd is huge (e.g. 100,000 users or more)*, tar
extractions can be excruciatingly slow. This is because,
although tar caches the last successful uid->username lookup
or username->uid lookup, it does not cache nonexistent user
names.

Thus, if /etc/passwd is huge and you have a tar file
containing many files owned by a user that is not on your
system, it will take forever to extract. (it will scan
through the entire /etc/passwd file for each file it
extracts)

The following patch makes tar extractions go about 100 times
faster in these cases, by caching non-existent user names,
group names, UIDs, and GIDs:

http://www.engin.umich.edu/caen/systems/Linux/code/patches/tar-1.12-faster.patch

(you should probably revert the "tar-1.12-namecache.patch"
in your Red Hat 6.0 tar RPM before applying this patch)


I have sent this patch to tar-bugs@gnu.ai.mit.edu, but if it
does not become part of the standard tar distribution, it
would be a good idea if it or something similar was included
in the next release of Red Hat Linux.

Thank you very much,

Chris Wing
wingc@engin.umich.edu



* - Yes, I know that Linux only supports 65,536 users
standard. I fixed that, too:

http://www.engin.umich.edu/caen/systems/Linux/highuids

Comment 1 Bill Nottingham 1999-07-09 16:45:59 UTC
Out of curiousity, does using nscd help at all?

Comment 2 wingc 1999-07-13 05:05:59 UTC
I haven't tried using nscd. Does nscd do any good if you aren't using
NIS?

In any case, I heard back from the maintainers of 'tar'-- the latest
version of GNU tar (1.13) contains a fix similar to mine.
Perhaps you should upgrade tar to v1.13 in the next version of Red Hat
Linux.

Comment 3 Bill Nottingham 1999-07-13 22:00:59 UTC
nscd might help some on huge passwd files; using db-based
passwd/group files would help more.

tar-1.13 will be in the next rawhide release.


Note You need to log in before you can comment on or make changes to this bug.