Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 483855 - Creating more than 66 easy curl handles causes a double free
Summary: Creating more than 66 easy curl handles causes a double free
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: nss
Version: 10
Hardware: All
OS: Linux
high
high
Target Milestone: ---
Assignee: Kai Engert (:kaie) (inactive account)
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 483222 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-02-03 22:47 UTC by Toshio Ernie Kuratomi
Modified: 2009-09-17 22:03 UTC (History)
9 users (show)

Fixed In Version: 3.12.2.0-3.fc9
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-04-13 19:37:24 UTC


Attachments (Terms of Use)
output from the crash (deleted)
2009-02-03 22:47 UTC, Toshio Ernie Kuratomi
no flags Details
Simple test case (deleted)
2009-02-03 23:26 UTC, Toshio Ernie Kuratomi
no flags Details
simple test in C (deleted)
2009-02-03 23:40 UTC, Toshio Ernie Kuratomi
no flags Details
program causing error within nss (deleted)
2009-02-06 10:30 UTC, Kamil Dudka
no flags Details
proposed patch (deleted)
2009-02-25 11:54 UTC, Kamil Dudka
no flags Details | Diff
same fix but patch leads to smaller code (deleted)
2009-02-28 14:33 UTC, Kamil Dudka
no flags Details | Diff

Description Toshio Ernie Kuratomi 2009-02-03 22:47:36 UTC
Created attachment 330791 [details]
output from the crash

Description of problem:
When using pycurl in the PackageDB, I eventually get a backtrace from glibc because of a double free.

Version-Release number of selected component (if applicable):
nspr-4.7.3-2.fc10.i386
nss-3.12.2.0-3.fc10.i386
python-pycurl-7.18.2-1.fc10.i386
curl-7.18.2-9.fc10.i386
python-2.5.2-1.fc10.i386

How reproducible:
In my application I can cause a crash over and over.  I haven't been able to generate a small test case yet, though.

  
Actual results:
Crash that glibc tells me is from a detected Double Free

Expected results:
Continue to make html requests to retrieve data.

Additional info:

Comment 1 Toshio Ernie Kuratomi 2009-02-03 23:26:11 UTC
Created attachment 330796 [details]
Simple test case

Okay, Here's a very simple test case that demonstrates how things are breaking.

In my testing, the crash occurs consistently on the 67th request.

Comment 2 Toshio Ernie Kuratomi 2009-02-03 23:40:42 UTC
Created attachment 330799 [details]
simple test in C

Just to make sure that the problem is in libcurl and not in pycurl, I wrote the test again in C.  Same results, curl causes a double free on the 67th invocation.

Comment 3 Kamil Dudka 2009-02-04 14:19:47 UTC
seems to be related to Bug 483222

Comment 4 Toshio Ernie Kuratomi 2009-02-04 17:06:03 UTC
(In reply to comment #3)
> seems to be related to Bug 483222

Could be. Do you know if git would be opening 67 curl objects in its lifetime?
(BTW, if git doesn't have to worry about thread safety, you might be able to workaround this and increase performance by only opening a single curl connection and making all requests through that.)

Comment 5 Toshio Ernie Kuratomi 2009-02-04 17:43:28 UTC
Testing with the simple test case found that: 

7.18.2 built from our SRPM without any of our patches is affected.
7.19.3 built from our SRPM without any of our patches is affected.

RHEL5's curl: curl-7.15.5-2.el5 is not affected.

Changing the configure options to --with-ssl=%{prefix} --without-nss::
    7.19.3 built from our SRPM without any of our patches is not affected
    7.18.2 with our patches is not affected

So -- this problem affects upstream as well.  It is linked to either nss or curl's use of nss, same as Bug 483222.

Comment 6 Jindrich Novy 2009-02-04 18:02:10 UTC
Well, it's more likely a NSS/NSPR fault I presume:

67 *** glibc detected *** ./a.out: double free or corruption (!prev): 0x0c7222a0 ***
======= Backtrace: =========
/lib/libc.so.6[0x3ba3a4]
/lib/libc.so.6(cfree+0x96)[0x3bc356]
/lib/libnspr4.so(PR_Free+0x27)[0x32f6b37]
/lib/libnsspem.so[0x2022f8]
/lib/libnsspem.so[0x1f079b]
/lib/libnsspem.so[0x1f23ce]
/lib/libnsspem.so[0x1f67f4]
/lib/libnsspem.so[0x1fe2c9]
/lib/libnsspem.so[0x1ed4ac]
/lib/libnss3.so[0x336aede]
/lib/libnss3.so(PK11_CreateGenericObject+0x50)[0x336b160]
/usr/lib/libcurl.so.4[0x1dc9a3]
/usr/lib/libcurl.so.4(Curl_nss_connect+0x747)[0x1dd3b7]
/usr/lib/libcurl.so.4(Curl_ssl_connect+0x37)[0x1d5017]
/usr/lib/libcurl.so.4(Curl_http_connect+0xa1)[0x1b4de1]
/usr/lib/libcurl.so.4(Curl_protocol_connect+0xe9)[0x1bbda9]
/usr/lib/libcurl.so.4(Curl_connect+0x314)[0x1bedc4]
/usr/lib/libcurl.so.4(Curl_perform+0x99)[0x1c9609]
/usr/lib/libcurl.so.4(curl_easy_perform+0x43)[0x1ca263]
./a.out[0x80486d3]

I had the same failure with either curl's downstream patches applied or not. Reassigning to NSS to get more info. Feel free to assign back or to nspr if needed.

Comment 7 Kamil Dudka 2009-02-06 10:30:35 UTC
Created attachment 331111 [details]
program causing error within nss

Indeed. Attaching minimal example. The output is:
...
61
62
63
==24187== Invalid write of size 4
==24187==    at 0x419E7BF: pem_CreateObject (pobject.c:1080)
==24187==    by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156)
==24187==    by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353)
==24187==    by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991)
==24187==    by 0x419B4AB: pemC_CreateObject (nssck.api:566)
==24187==    by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412)
==24187==    by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346)
==24187==    by 0x804873A: main (in /tmp/curl/a.out)
==24187==  Address 0x8a2d578 is 0 bytes after a block of size 63,496 alloc'd
==24187==    at 0x4004BA2: calloc (vg_replace_malloc.c:397)
==24187==    by 0x703E039: PR_Calloc (in /lib/libnspr4.so)
==24187==    by 0x41B0297: nss_ZRealloc (arena.c:1056)
==24187==    by 0x419E79A: pem_CreateObject (pobject.c:1080)
==24187==    by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156)
==24187==    by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353)
==24187==    by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991)
==24187==    by 0x419B4AB: pemC_CreateObject (nssck.api:566)
==24187==    by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412)
==24187==    by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346)
==24187==    by 0x804873A: main (in /tmp/curl/a.out)

Comment 8 Kamil Dudka 2009-02-25 11:54:43 UTC
Created attachment 333153 [details]
proposed patch

It's bug in pem/pobject.c, kind of +/-1 problem. Fix attached.

Comment 9 Kamil Dudka 2009-02-25 11:55:56 UTC
*** Bug 483222 has been marked as a duplicate of this bug. ***

Comment 10 Toshio Ernie Kuratomi 2009-02-25 16:40:42 UTC
A rebuilt nss with the supplied patch no longer crashes.  However, it either causes or allows a memory leak to surface.  Running any of the three test programs runs my 1GB machine into swap.

The simple curl test cases run me into swap before we hit 200 iterations.  The nss test case before we hit 2000 iterations.

Comment 11 Kamil Dudka 2009-02-26 10:44:04 UTC
I am sure there are no leaks caused by the patch. Looking at the nss test, there is no stop condition, no free, no destroy, no shutdown, ... The test was intended to be minimal, not to free all resources. If you think there is a leak within nss or curl please open separate bug for it.

Slightly raising priority as the bug is obvious and patch is simple.

Comment 12 Kamil Dudka 2009-02-28 14:33:52 UTC
Created attachment 333603 [details]
same fix but patch leads to smaller code

Comment 13 Toshio Ernie Kuratomi 2009-03-05 16:32:50 UTC
The new patch works here as well.  Something was not right in how I tested for the memory leak before... I no longer leak memory in the C curl test so it seems like an issue with the python bindings.  I'll report a bug there after I gather more info.

Comment 14 Kamil Dudka 2009-03-11 10:34:58 UTC
reported upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=482678

Comment 15 Kamil Dudka 2009-03-11 17:15:53 UTC
reported again, now the right place hopefully :-)
https://bugzilla.mozilla.org/show_bug.cgi?id=402712

Comment 16 Kamil Dudka 2009-03-16 09:07:07 UTC
Ready to patch Fedora?

Comment 17 Kai Engert (:kaie) (inactive account) 2009-03-31 23:01:42 UTC
Kamil, thanks for producing the patch and getting it reviewed upstream.

Comment 18 Kai Engert (:kaie) (inactive account) 2009-04-01 13:23:37 UTC
I produced a new NSS build with the patch applied. It's not in the update channel yet but can be downloaded from this page:

https://koji.fedoraproject.org/koji/taskinfo?taskID=1268901

For example click i386 which brings you to:
https://koji.fedoraproject.org/koji/taskinfo?taskID=1268907

Which has links to the .rpm files.

Could you please test this build and report whether it fixes your problem?
Thanks!

...
the above build is for fedora 10
f9 build: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268909
rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268884

Comment 19 Toshio Ernie Kuratomi 2009-04-06 21:31:47 UTC
Tested.  The build fixes the crash.  The memory leak is a separate issue that Kamil is talking to upstream about.

Comment 20 Joerg (kital) Simon 2009-04-09 17:54:53 UTC
this fixed Version NSS solved my problem with a 
error:
*** glibc detected *** python: double free or corruption (!prev): 0x0af39ee0 ***
from the script https://fedorahosted.org/fama/browser/fas-group-inactive.py

Comment 21 Fedora Update System 2009-04-10 01:21:11 UTC
nss-3.12.2.0-5.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/nss-3.12.2.0-5.fc10

Comment 22 Fedora Update System 2009-04-10 01:21:17 UTC
nss-3.12.2.0-3.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nss-3.12.2.0-3.fc9

Comment 23 Kai Engert (:kaie) (inactive account) 2009-04-10 01:23:02 UTC
Thanks for the test feedback.

I've pushed the updated packages to the fedora update system.

Comment 24 Fedora Update System 2009-04-13 19:37:18 UTC
nss-3.12.2.0-5.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2009-04-13 19:44:11 UTC
nss-3.12.2.0-3.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 26 Bryan Field-Elliot 2009-05-18 20:15:56 UTC
I would be very grateful if someone could build and publish this update to nss for Fedora 8. If it's already been done, then perhaps someone could tell me where to find it? Thank you.

Comment 27 Kamil Dudka 2009-05-18 20:32:08 UTC
Fedora 8 is no longer supported, so there are AFAIK no updates. Does a scratch build do the job for you?

Comment 28 Kai Engert (:kaie) (inactive account) 2009-05-19 21:55:50 UTC
Bryan, as Kamil said, we are no longer able to build updates for you.

Get the Fedora 8 source RPM, find documentation about building from a source RPM, add the patch we added in Fedora 9 and above, and build it.

Patch is attached in this bug, attachment 333603 [details]

Comment 29 Kai Engert (:kaie) (inactive account) 2009-05-19 21:57:30 UTC
http://fedoraproject.org/wiki/LifeCycle
http://lwn.net/Articles/308913/

Comment 30 Kamil Dudka 2009-05-19 22:55:38 UTC
Well, considering severity of the bug I've made the scratch build anyway:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1364823

Comment 31 Bryan Field-Elliot 2009-05-23 16:14:46 UTC
Kamil, thanks very much for building this update to nss for FC8. I've installed the binary RPM and it looks like it fixed the problem for me. 

Thank you again,

Bryan

Comment 32 Fedora Update System 2009-09-16 04:09:04 UTC
nss-3.12.4-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/nss-3.12.4-2.fc11

Comment 33 Fedora Update System 2009-09-17 22:03:35 UTC
nss-3.12.4-3.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/nss-3.12.4-3.fc11?_csrf_token=d6eee54945d598c83d89d6481b2a20cf972bc76b


Note You need to log in before you can comment on or make changes to this bug.