Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 822632 - ValueError: need more than 3 values to unpack
Summary: ValueError: need more than 3 values to unpack
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-urlgrabber
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Fedora Packaging Toolset Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-05-17 16:23 UTC by Frank Murphy
Modified: 2014-01-21 23:22 UTC (History)
7 users (show)

Fixed In Version: python-urlgrabber-3.9.1-14.fc18
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-13 09:21:23 UTC


Attachments (Terms of Use)
timerhosts.txt (deleted)
2012-05-18 07:58 UTC, Frank Murphy
no flags Details
timedhosts (deleted)
2012-05-18 07:59 UTC, Frank Murphy
no flags Details
URLGRABBER_DEBUG=1,debug.txt yum update (deleted)
2012-05-18 10:45 UTC, Frank Murphy
no flags Details

Description Frank Murphy 2012-05-17 16:23:25 UTC
Description of problem: Exits with Traceback


Version-Release number of selected component (if applicable):
python-urlgrabber-3.9.1-12.fc18.noarch
yum-3.4.3-26.fc18.noarch

How reproducible: Number times, both Packagekit using softwarte install from:
http://kojipkgs.fedoraproject.org/packages/kernel/3.4.0/0.rc7.git3.1.fc18/i686/kernel-3.4.0-0.rc7.git3.1.fc18.i686.rpm

and yum as follows (latest traceback):


yum clean metadata && createrepo -v /nfs/yum/rawhide/i386/_local/packages && yum update -y --disableplugin=fastestmirror

Traceback (most recent call last):
  File "/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 321, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 146, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 485, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 405, in doCommand
    return base.updatePkgs(extcmds, update_to=(basecmd == 'update-to'))
  File "/usr/share/yum-cli/cli.py", line 890, in updatePkgs
    self.update()
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4363, in update
    updates = self.up.getUpdatesTuples()
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1031, in <lambda>
    up = property(fget=lambda self: self._getUpdates(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 836, in _getUpdates
    self._up = rpmUtils.updates.Updates(self.rpmdb.simplePkgList(), self.pkgSack.simplePkgList())
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1012, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 776, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 299, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 125, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 81, in retrieveAllMD
    if repo._async and repo._commonLoadRepoXML(repo):
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1329, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1107, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 928, in _getFile
    **kwargs
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 425, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 399, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1125, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1021, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1116, in retryfunc
    _TH.update(url, fo._amount_read - fo._reget_length, time.time() - tm, None)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2292, in update
    _TH.load()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2271, in load
    host, speed, fail, ts = line.split()
ValueError: need more than 3 values to unpack

Comment 1 Zdeněk Pavlas 2012-05-18 07:37:03 UTC
Thanks for the report!  Seems /var/cache/yum/timedhosts file got malformed somehow (empty line, less than 4 values on some line etc).  Could you attach it, please?  It's probably an easy fix, but I'd like to know the root cause, before just ignoring such lines when reading mirror speed info.

Comment 2 Frank Murphy 2012-05-18 07:58:36 UTC
Created attachment 585358 [details]
timerhosts.txt

Comment 3 Frank Murphy 2012-05-18 07:59:50 UTC
Created attachment 585359 [details]
timedhosts

Also found this in same dir, removing this no difference to traceback.

Comment 4 Zdeněk Pavlas 2012-05-18 09:18:08 UTC
Thanks!

' 22165 0 1337268949' is the problem, the "hostname" field is empty.  Curiously enough the mirror does not seem to be failing.

You probably have some bogus baseurl set up in some repo, eg http:///foo.org (note 3 slashes), but it might also come from some mirrorlist, can't tell.
Could be a bug in urlparse module, too.  I'll add a sanity check for this.

You might want to run Yum again with "export URLGRABBER_DEBUG=1,debug.txt", and check URLs in debug.txt, or attach the log to the BZ.

Comment 5 Frank Murphy 2012-05-18 10:09:17 UTC
(In reply to comment #4)
> Thanks!

> 
> You probably have some bogus baseurl set up in some repo, eg http:///foo.org
> (note 3 slashes), but it might also come from some mirrorlist, can't tell.
> Could be a bug in urlparse module, too.  I'll add a sanity check for this.

Check /etc/yum.repos.d/*.repo
no /// in any.

> 
> You might want to run Yum again with "export URLGRABBER_DEBUG=1,debug.txt", and
> check URLs in debug.txt, or attach the log to the BZ.

Where do I place this exactly tried:
"yum update export URLGRABBER_DEBUG=1,debug.txt"
or does it go in /etc/yum.conf

but cannot find debug.txt, even after updatedb

Comment 6 Zdeněk Pavlas 2012-05-18 10:28:46 UTC
Hi,

"URLGRABBER_DEBUG=1,debug.txt yum update" should work.  Also, you will have to "rm /var/cache/yum/timedhosts", otherwise Yum will crash early.

Comment 7 Frank Murphy 2012-05-18 10:45:07 UTC
Created attachment 585387 [details]
URLGRABBER_DEBUG=1,debug.txt yum update

Comment 8 Frank Murphy 2012-05-18 11:01:16 UTC
Got the cause of the problem,
Cache for all Fedora release is ketp at:
cachedir=/nfs/yum/$releasever/$basearch
(saves online bandwidth)

Somehow the rawhide ba was pulling timeshosts from:
/nfs/yum/18/i386/timedhosts*

instead of:
/nfs/yum/rawhide/i386
which has no timedhosts* as I deleted them before running dubug.

This bug  be particular to my setup.
though local.conf, yum.conf, _local.repo are set correctly for rawhide.

Comment 9 Zdeněk Pavlas 2012-05-18 11:49:43 UTC
I've managed to reproduce this myself.  It's the 'file://' URL handling..  These are normally handled immediately and NOT profiled, except for when such URLs are inside a mirrorlist (then we crash).  There's a quick fix:

--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -2291,6 +2291,8 @@ class _TH:
     def update(url, dl_size, dl_time, ug_err):
         _TH.load()
         host = urlparse.urlsplit(url).netloc
+        if not host: # file://
+            return
         speed, fail, ts = _TH.hosts.get(host) or (0, 0, 0)
         now = time.time()

But I think that file:// mirrors SHOULD be profiled, too, so this bug needs a more complex fix.  Also, urlspilt() does not strip USER:PASS@ prefix so this ends up in timedhosts (ouch), have to fix this, too.

Comment 10 Frank Murphy 2012-05-18 12:04:35 UTC
It's not particular to me so.  Yippee, I'm not mad :)

Comment 11 Bruno Wolff III 2012-05-18 12:22:11 UTC
I am seeing it too. That's why I added myself as a cc last night.

Comment 12 Frank Murphy 2012-05-18 14:54:32 UTC
python-urlgrabber-3.9.1-13.fc18 works for me.

Comment 13 Remi Collet 2012-05-18 16:08:22 UTC
with
python-urlgrabber-3.9.1-13.fc18.noarch
yum-3.4.3-26.fc18.noarch

Works only "once", on second call 

Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 321, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 146, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 485, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 405, in doCommand
    return base.updatePkgs(extcmds, update_to=(basecmd == 'update-to'))
  File "/usr/share/yum-cli/cli.py", line 890, in updatePkgs
    self.update()
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4363, in update
    updates = self.up.getUpdatesTuples()
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1031, in <lambda>
    up = property(fget=lambda self: self._getUpdates(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 836, in _getUpdates
    self._up = rpmUtils.updates.Updates(self.rpmdb.simplePkgList(), self.pkgSack.simplePkgList())
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1012, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 776, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 299, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 125, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 94, in retrieveAllMD
    urlgrabber.grabber.parallel_wait()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2198, in parallel_wait
    perform()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2145, in perform
    for opts, size, ug_err in dl.perform():
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2093, in perform
    done = self.running[fd].perform()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2063, in perform
    _TH.update(opts.url, int(line[2]), float(line[3]), ug_err, opts.async[0])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2308, in update
    speed = (k1 * speed + k2 * dl_size / dl_time) / (k1 + k2)
ZeroDivisionError: float division by zero

in timedhosts
  file:/home/rpmbuild/site/rpms/fedora/17/remi/i386/ 438764923 0 1337357074
  ftp.free.fr 52873 0 1337357084
  ftp.proxad.net 5146 0 1337357084

Comment 14 Ben Boeckel 2012-05-19 16:12:47 UTC
I'm seeing this as well.

> file:/home/rpmbuild/site/rpms/fedora/17/remi/i386/

I believe that this is an invalid URI. It should be file:///home/...

Comment 15 Remi Collet 2012-05-20 16:17:20 UTC
Same error with file:///home uri.

Adding the trivial test, is a workaround.
    if (dl_time > 0) and ((k1+k2) > 0):
        speed = (k1 * speed + k2 * dl_size / dl_time) / (k1 + k2)

Comment 16 Zdeněk Pavlas 2012-05-21 07:58:42 UTC
> Works only "once", on second call

Apparently, local "file://" mirrors are infinitely fast once the files got buffered :)  Added a sanity check and changed the speed averaging in python-urlgrabber-3.9.1-14.fc18

Comment 17 Remi Collet 2012-05-21 11:29:05 UTC
python-urlgrabber-3.9.1-14.fc18 seems ok (cannot reproduce the error)
Thanks for the fix.

Comment 18 Fedora End Of Life 2013-04-03 20:29:31 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 19 Ben Boeckel 2013-06-13 03:39:09 UTC
This should probably be closed as FIXED. I haven't seen this bug in ages (Rawhide, 19 (though no file:// repos there), and 18).


Note You need to log in before you can comment on or make changes to this bug.