Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1066335 - [perf] population of metadata cache
Summary: [perf] population of metadata cache
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 20
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-18 09:20 UTC by Tim Lauridsen
Modified: 2014-04-02 12:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-04-02 12:00:19 UTC


Attachments (Terms of Use)

Description Tim Lauridsen 2014-02-18 09:20:00 UTC
Description of problem:

When dnf was to populate the metadata cache it takes around 4 min
yum uses 45 sec todo the same,


Version-Release number of selected component (if applicable):

0.4.13

How reproducible:
every time


Steps to Reproduce:
1. sudo dnf clean aææ
2. time sudo dnf makecache
3.

Actual results:
Zzzzzzzz :)


Expected results:
As fast as yum

If dnf shall be a replacement for yum, it need to been as fast as yum

I know this is a worst case senario, and you dont do a full cache refresh every
time.


Additional info:

$ sudo dnf clean all
$ time sudo dnf makecache
Fedora 20 - x86_64                              5.4 MB/s |  36 MB     00:06    
RPM Fusion for Fedora 20 - Free - Updates       895 kB/s | 182 kB     00:00    
RPM Fusion for Fedora 20 - Nonfree - Updates    467 kB/s |  55 kB     00:00    
RPM Fusion for Fedora 20 - Free                 1.1 MB/s | 487 kB     00:00    
Fedora 20 - x86_64 - Updates                    141 kB/s |  17 MB     02:02    
google-chrome                                    43 kB/s | 3.1 kB     00:00    
RPM Fusion for Fedora 20 - Nonfree              1.3 MB/s | 289 kB     00:00    
Copr repo for yumex-nextgen owned by timlau     4.4 kB/s | 1.7 kB     00:00    
Metadata cache created.

real	3m48.178s
user	0m23.758s
sys	0m3.750s

$ sudo yum clean all
$ time sudo yum makecache
Loaded plugins: langpacks
fedora/20/x86_64/metalink                                |  34 kB     00:00     
fedora                                                   | 3.8 kB     00:00     
google-chrome                                            |  951 B     00:00     
rpmfusion-free                                           | 3.3 kB     00:00     
rpmfusion-free-updates                                   | 3.3 kB     00:00     
rpmfusion-nonfree                                        | 3.3 kB     00:00     
rpmfusion-nonfree-updates                                | 3.3 kB     00:00     
timlau-yumex-nextgen                                     | 3.0 kB     00:00     
updates/20/x86_64/metalink                               |  26 kB     00:00     
updates                                                  | 4.6 kB     00:00     
(1/12): fedora/20/x86_64/group_gz                          | 394 kB   00:00     
(2/12): fedora/20/x86_64/filelists_db                      |  27 MB   00:04     
(3/12): fedora/20/x86_64/primary_db                        |  18 MB   00:03     
(4/12): timlau-yumex-nextgen/20/x86_64/filelists_db        | 1.8 kB   00:00     
(5/12): timlau-yumex-nextgen/20/x86_64/primary_db          | 2.4 kB   00:00     
(6/12): timlau-yumex-nextgen/20/x86_64/other_db            |  842 B   00:00     
(7/12): updates/20/x86_64/group_gz                         | 394 kB   00:00     
(8/12): fedora/20/x86_64/other_db                          | 8.7 MB   00:03     
(9/12): updates/20/x86_64/filelists_db                     |  12 MB   00:03     
(10/12): updates/20/x86_64/prestodelta                     | 1.5 MB   00:00     
(11/12): updates/20/x86_64/primary_db                      | 7.9 MB   00:02     
(12/12): updates/20/x86_64/other_db                        | 4.5 MB   00:03     
(1/21): google-chrome/primary                              | 1.9 kB   00:00     
(2/21): google-chrome/filelists                            | 1.2 kB   00:00     
(3/21): google-chrome/other                                |  448 B   00:00     
(4/21): rpmfusion-free/20/x86_64/group_gz                  | 1.6 kB   00:00     
(5/21): rpmfusion-free/20/x86_64/filelists_db              | 325 kB   00:00     
(6/21): rpmfusion-free/20/x86_64/other_db                  | 111 kB   00:00     
(7/21): rpmfusion-free/20/x86_64/primary_db                | 445 kB   00:00     
(8/21): rpmfusion-free-updates/20/x86_64/group_gz          | 1.6 kB   00:00     
(9/21): rpmfusion-free-updates/20/x86_64/filelists_db      |  95 kB   00:00     
(10/21): rpmfusion-free-updates/20/x86_64/other_db         |  40 kB   00:00     
(11/21): rpmfusion-free-updates/20/x86_64/primary_db       | 152 kB   00:00     
(12/21): rpmfusion-nonfree/20/x86_64/group_gz              | 1.0 kB   00:00     
(13/21): rpmfusion-nonfree/20/x86_64/filelists_db          | 218 kB   00:00     
(14/21): rpmfusion-nonfree/20/x86_64/primary_db            | 137 kB   00:00     
(15/21): rpmfusion-nonfree/20/x86_64/other_db              |  54 kB   00:00     
(16/21): rpmfusion-nonfree-updates/20/x86_64/group_gz      | 1.0 kB   00:00     
(17/21): rpmfusion-nonfree-updates/20/x86_64/filelists_db  |  31 kB   00:00     
(18/21): rpmfusion-nonfree-updates/20/x86_64/other_db      |  25 kB   00:00     
(19/21): rpmfusion-nonfree-updates/20/x86_64/primary_db    |  53 kB   00:00     
(20/21): updates/20/x86_64/updateinfo                      | 748 kB   00:00     
(21/21): updates/20/x86_64/pkgtags                         | 967 kB   00:00     
google-chrome                                                               3/3
google-chrome                                                               3/3
google-chrome                                                               3/3
Metadata Cache Created

real	0m44.089s
user	0m21.377s
sys	0m1.260s

Comment 1 Ales Kozumplik 2014-02-18 10:49:26 UTC
> If dnf shall be a replacement for yum, it need to been as fast as yum

No and at it already is for the most frequent scenarios.

I am not even certain at this point large speedups can be gained---we just deal with that much data in libsolv.

Comment 2 Jan Zeleny 2014-02-18 13:22:17 UTC
Let me elaborate a bit. The underlying cause here is that yum does very little (if anything) to create the cache, because it essentially downloads everything from remote repos. This approach then leads to some slowdown during dependency resolution when you perform it.

Dnf on the other hand constructs some special structures that later allow it to process dependencies much faster. And if I understand it correctly, that's what takes forever on your machine. Therefore as Ales said, there is a tradeoff - dnf trades some time spent on metadata creation for a lot of time saved solving dependencies.

Comment 3 Tim Lauridsen 2014-04-02 12:00:19 UTC
closing this one, I can see the differnt approch dnf is using.
the cache is not updated so dynamic as yum, so it is not a issue in everyday use


Note You need to log in before you can comment on or make changes to this bug.