Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 88399 - segfault in /lib/libpthread.so.0
Summary: segfault in /lib/libpthread.so.0
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: glibc
Version: 8.0
Hardware: athlon
OS: Linux
medium
high
Target Milestone: ---
Assignee: Jakub Jelinek
QA Contact: Brian Brock
URL: http://p1.selectacast.net/~jks/hs_err...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-04-09 19:38 UTC by Joseph Shraibman
Modified: 2016-11-24 15:16 UTC (History)
1 user (show)

Fixed In Version: 2.3.2-4.80.6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2003-10-08 22:57:17 UTC


Attachments (Terms of Use)

Description Joseph Shraibman 2003-04-09 19:38:56 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4a) Gecko/20030401

Description of problem:
When running a java program that uses a lot of threads (up to 3000ish) I get this:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x4002CFEC
Function=(null)+0x4002CFEC
Library=/lib/libpthread.so.0

NOTE: We are unable to locate the function name symbol for the error
      just occurred. Please refer to release documentation for possible
      reason and solutions.


Current Java thread:
	at java.lang.Thread.start(Native Method)
	- locked <0x4578a288> (a java.lang.Thread)
	at org.xbill.DNS.SimpleResolver.sendAsync(SimpleResolver.java:315)
	at org.xbill.DNS.ExtendedResolver.sendTo(ExtendedResolver.java:148)
	- locked <0x45789ed0> (a java.util.HashMap)
	at org.xbill.DNS.ExtendedResolver.send(ExtendedResolver.java:248)
	- locked <0x45789ea0> (a java.util.LinkedList)
	at org.xbill.DNS.Lookup.lookup(Lookup.java:384)
	at org.xbill.DNS.Lookup.resolve(Lookup.java:424)
	at org.xbill.DNS.Lookup.run(Lookup.java:441)
	at org.xbill.DNS.dns.getRecords(dns.java:146)
	at org.xbill.DNS.dns.getAnyRecords(dns.java:195)
        <snip>


Dynamic libraries:
08048000-0804e000 r-xp 00000000 03:03 2245061    /usr/local/j2sdk1.4.1_02/bin/java
0804e000-0804f000 rw-p 00005000 03:03 2245061    /usr/local/j2sdk1.4.1_02/bin/java
40000000-40012000 r-xp 00000000 03:03 376907     /lib/ld-2.3.2.so
40012000-40013000 rw-p 00011000 03:03 376907     /lib/ld-2.3.2.so
40013000-4001c000 r-xp 00000000 03:03 1655143   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/native_threads/libhpi.so
4001c000-4001d000 rw-p 00008000 03:03 1655143   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/native_threads/libhpi.so
4001e000-4001f000 r--p 00a08000 03:03 426253     /usr/lib/locale/locale-archive
4001f000-40022000 r--s 00000000 03:03 2457778   
/usr/local/j2sdk1.4.1_02/jre/lib/ext/dnsns.jar
40022000-40030000 r-xp 00000000 03:03 376926     /lib/libpthread-0.10.so
40030000-40033000 rw-p 0000e000 03:03 376926     /lib/libpthread-0.10.so
40074000-40076000 r-xp 00000000 03:03 376857     /lib/libdl-2.3.2.so
40076000-40077000 rw-p 00001000 03:03 376857     /lib/libdl-2.3.2.so
40077000-40199000 r-xp 00000000 03:03 376854     /lib/libc-2.3.2.so
40199000-4019e000 rw-p 00121000 03:03 376854     /lib/libc-2.3.2.so
401a1000-4063a000 r-xp 00000000 03:03 2097204   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/server/libjvm.so
4063a000-40872000 rw-p 00498000 03:03 2097204   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/server/libjvm.so
<rest snipped becuse if I don't apache complains that the request uri is too long>

After that error appears the jvm doesn't core dump or go away, it just sits
there.  Existing rmi calls just never return and new ones result in:
java.rmi.ConnectIOException: error during JRMP connection establishment; nested
exception is: java.net.SocketTimeoutException: Read timed out

Version-Release number of selected component (if applicable):
glibc-2.3.2-4.80

How reproducible:
Sometimes

Steps to Reproduce:
Seems hard to reproduce.  It seems to require a lot of threads and high memory usage

    

Additional info:

redhat 8.0 dual athlon system
java version 1.4.1_02 from java.sun.com

We never got this crash before the recent glibc upgrade, but that doesn't mean
anything. This crash also occured with jvm 1.4.1_01. The crash occurs when
spinning off a thread. We have had this crash with in different parts of the
code that spin off threads.  It occurs with client and server jvms.

Comment 1 Jakub Jelinek 2003-04-09 19:46:13 UTC
Please try ftp://people.redhat.com/jakub/glibc/errata/8.0/
2.3.2-4.80 because of a kernel bug doesn't use /lib/i686/libpthread.so.0
but /lib/libpthread.so.0 (that is a compatibility with old kernels version,
with fixed size 2MB thread stacks, so with 3000 * 2M you might very well
reach address space limits (libraries are mapped at around 1GB, kernel starts
at 3GB and thread stack array is in between).

Comment 2 Joseph Shraibman 2003-04-09 20:18:42 UTC
What is the kernel bug?  Will I run into it if I install these rpms?  Do I need
to reboot my machine after this?

Comment 3 Joseph Shraibman 2003-04-10 02:40:17 UTC
And is this fixed in redhat 9?

Comment 4 Jakub Jelinek 2003-04-10 22:44:56 UTC
The kernel bug is that AT_PLATFORM aux vector is passed wrong value to the
applications by the kernel. Try LD_SHOW_AUXV=1 /bin/echo, AT_PLATFORM
should be i686 on i686+ box, but 8.0 kernel passes an empty string.
This is fixed in RHL9 kernel and hopefully will be fixed in next 8.0 errata
kernel too when/if that happens.
As for the glibc errata, 2.3.2-4.80-6 will be pushed officially as errata
today.

Comment 5 Joseph Shraibman 2003-04-14 18:42:59 UTC
Upgrading to glibc-2.3.2-4.80.6 solved this problem.

Comment 6 Joseph Shraibman 2003-10-08 03:55:57 UTC
I just upgraded a system to redhat 9 and this bug seems to have regressed.

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x4002CB28
Function=(null)+0x4002CB28
Library=/lib/libpthread.so.0

NOTE: We are unable to locate the function name symbol for the error
      just occurred. Please refer to release documentation for possible
      reason and solutions.


Current Java thread:
        at java.lang.Thread.start(Native Method)
        - locked <0x44cd3df8> (a java.lang.Thread)
        at com.xtenit.mail.DeliverySystem2Imp.deliver0(DeliverySystem2Imp.java:923)
        at
com.xtenit.mail.DeliverySystem2Imp.deliver0wrapper(DeliverySystem2Imp.java:496)
        at com.xtenit.mail.DeliverySystem2Imp.deliver(DeliverySystem2Imp.java:481)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
        at sun.rmi.transport.Transport$1.run(Transport.java:148)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
        at java.lang.Thread.run(Thread.java:534)

Dynamic libraries:
08048000-0804e000 r-xp 00000000 03:03 2736898    /usr/local/j2sdk1.4.2/bin/java
0804e000-0804f000 rw-p 00005000 03:03 2736898    /usr/local/j2sdk1.4.2/bin/java
40000000-40012000 r-xp 00000000 03:03 376922     /lib/ld-2.3.2.so
40012000-40013000 rw-p 00011000 03:03 376922     /lib/ld-2.3.2.so
40014000-4001c000 r-xp 00000000 03:03 1294916   
/usr/local/j2sdk1.4.2/jre/lib/i386/native_threads/libhpi.so
4001c000-4001d000 rw-p 00007000 03:03 1294916   
/usr/local/j2sdk1.4.2/jre/lib/i386/native_threads/libhpi.so
4001d000-40021000 rw-s 00000000 03:03 868361     /tmp/hsperfdata_runner/22448
40021000-40022000 r--p 00a08000 03:03 425990     /usr/lib/locale/locale-archive
40022000-40030000 r-xp 00000000 03:03 376917     /lib/libpthread-0.10.so
40030000-40033000 rw-p 0000e000 03:03 376917     /lib/libpthread-0.10.so
40073000-40075000 r-xp 00000000 03:03 376860     /lib/libdl-2.3.2.so
40075000-40076000 rw-p 00002000 03:03 376860     /lib/libdl-2.3.2.so
40076000-40196000 r-xp 00000000 03:03 376837     /lib/libc-2.3.2.so
40196000-40199000 rw-p 00120000 03:03 376837     /lib/libc-2.3.2.so
4019c000-40748000 r-xp 00000000 03:03 1294918   
/usr/local/j2sdk1.4.2/jre/lib/i386/server/libjvm.so
40748000-407a2000 rw-p 005ab000 03:03 1294918   
/usr/local/j2sdk1.4.2/jre/lib/i386/server/libjvm.so
407b5000-407c5000 r-xp 00000000 03:03 376868     /lib/libnsl-2.3.2.so
407c5000-407c6000 rw-p 00010000 03:03 376868     /lib/libnsl-2.3.2.so
407c8000-407e8000 r-xp 00000000 03:03 376866     /lib/libm-2.3.2.so
407e8000-407e9000 rw-p 00020000 03:03 376866     /lib/libm-2.3.2.so
407e9000-407ec000 r--s 00000000 03:03 3801317   
/usr/local/j2sdk1.4.2/jre/lib/ext/dnsns.jar
407ec000-407f4000 r-xp 00000000 03:03 376907     /lib/libnss_nis-2.3.2.so
407f4000-407f5000 rw-p 00007000 03:03 376907     /lib/libnss_nis-2.3.2.so
407f5000-407f6000 r-xp 00000000 03:03 1098823   
/usr/local/j2sdk1.4.2/jre/lib/i386/librmi.so
407f6000-407f7000 rw-p 00000000 03:03 1098823   
/usr/local/j2sdk1.4.2/jre/lib/i386/librmi.so
407f7000-40801000 r-xp 00000000 03:03 376893     /lib/libnss_files-2.3.2.so
40801000-40802000 rw-p 00009000 03:03 376893     /lib/libnss_files-2.3.2.so
40802000-40812000 r-xp 00000000 03:03 1098805   
/usr/local/j2sdk1.4.2/jre/lib/i386/libverify.so
40812000-40814000 rw-p 0000f000 03:03 1098805   
/usr/local/j2sdk1.4.2/jre/lib/i386/libverify.so
40814000-40834000 r-xp 00000000 03:03 1098806   
/usr/local/j2sdk1.4.2/jre/lib/i386/libjava.so
40834000-40836000 rw-p 0001f000 03:03 1098806   
/usr/local/j2sdk1.4.2/jre/lib/i386/libjava.so
40836000-4084a000 r-xp 00000000 03:03 1098808   
/usr/local/j2sdk1.4.2/jre/lib/i386/libzip.so
4084a000-4084d000 rw-p 00013000 03:03 1098808   
/usr/local/j2sdk1.4.2/jre/lib/i386/libzip.so
4084d000-421d8000 r--s 00000000 03:03 1115216   
/usr/local/j2sdk1.4.2/jre/lib/rt.jar
42222000-42238000 r--s 00000000 03:03 1114890   
/usr/local/j2sdk1.4.2/jre/lib/sunrsasign.jar
42238000-42313000 r--s 00000000 03:03 1115213   
/usr/local/j2sdk1.4.2/jre/lib/jsse.jar
42313000-42324000 r--s 00000000 03:03 1114921   
/usr/local/j2sdk1.4.2/jre/lib/jce.jar
42324000-4287d000 r--s 00000000 03:03 1115214   
/usr/local/j2sdk1.4.2/jre/lib/charsets.jar
44925000-4492e000 r--s 00000000 03:03 1081522    /usr/local/java/classes/cos.jar
78c48000-78e48000 r--p 00000000 03:03 425990     /usr/lib/locale/locale-archive
78e48000-78e7a000 r--p 009ba000 03:03 425990     /usr/lib/locale/locale-archive
78e7a000-78e96000 r--s 00000000 03:03 3801316   
/usr/local/j2sdk1.4.2/jre/lib/ext/sunjce_provider.jar
78e96000-78ea3000 r--s 00000000 03:03 3801318   
/usr/local/j2sdk1.4.2/jre/lib/ext/ldapsec.jar
78ea3000-78f5f000 r--s 00000000 03:03 3801319   
/usr/local/j2sdk1.4.2/jre/lib/ext/localedata.jar
78f5f000-78faa000 r--s 00000000 03:03 1081481    /usr/local/java/classes/smtp.jar
78faa000-78fba000 r-xp 00000000 03:03 1098811   
/usr/local/j2sdk1.4.2/jre/lib/i386/libnet.so
78fba000-78fbb000 rw-p 0000f000 03:03 1098811   
/usr/local/j2sdk1.4.2/jre/lib/i386/libnet.so
78fbb000-78fc9000 r--s 00000000 03:03 1081451   
/usr/local/java/classes/activation.102.jar
78fc9000-78fe5000 r--s 00000000 03:03 1081774   
/usr/local/java/classes/xml-apis.xalan241.jar
78fe5000-78ffb000 r--s 00000000 03:03 1081514   
/usr/local/java/classes/NetComponents.jar
78ffb000-78ffe000 r-xp 00000000 03:03 376878     /lib/libnss_dns-2.3.2.so
78ffe000-78fff000 rw-p 00002000 03:03 376878     /lib/libnss_dns-2.3.2.so
79100000-79150000 r--s 00000000 03:03 1081779   
/usr/local/java/classes/mail-1.3.1ea.jar
79150000-7921c000 r--s 00000000 03:03 1081444   
/usr/local/java/classes/xerces.221.jar
7921c000-79318000 r--s 00000000 03:03 1081773   
/usr/local/java/classes/xalan.241.jar
79318000-79334000 r--s 00000000 03:03 1081774   
/usr/local/java/classes/xml-apis.xalan241.jar
79334000-79370000 r--s 00000000 03:03 1081780   
/usr/local/java/classes/bsh-1.3b2.jar
79370000-7937d000 r--s 00000000 03:03 1081546    /usr/local/java/classes/mbox.jar
7937d000-79385000 r--s 00000000 03:03 1081543   
/usr/local/java/classes/gnu-regexp-1.1.2.jar
79385000-793b3000 r--s 00000000 03:03 1081776   
/usr/local/java/classes/dnsjava-1.3.2.jar
793c1000-793ce000 r-xp 00000000 03:03 376926     /lib/libresolv-2.3.2.so
793ce000-793cf000 rw-p 0000d000 03:03 376926     /lib/libresolv-2.3.2.so

Heap at VM Abort:
Heap
 def new generation   total 8384K, used 5549K [0x44930000, 0x45240000, 0x49e80000)
  eden space 7488K,  62% used [0x44930000, 0x44dbb600, 0x45080000)
  from space 896K, 100% used [0x45080000, 0x45160000, 0x45160000)
  to   space 896K,   0% used [0x45160000, 0x45160000, 0x45240000)
 tenured generation   total 50556K, used 37999K [0x49e80000, 0x4cfdf000, 0x74930000)
   the space 50556K,  75% used [0x49e80000, 0x4c39bde0, 0x4c39be00, 0x4cfdf000)
 compacting perm gen  total 16384K, used 7293K [0x74930000, 0x75930000, 0x78930000)
   the space 16384K,  44% used [0x74930000, 0x7504f428, 0x7504f600, 0x75930000)

Local Time = Tue Oct  7 22:52:54 2003
Elapsed Time = 36848
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2-b28 mixed mode)
#
# An error report file has been saved as hs_err_pid22448.log.
# Please refer to the file for further information.
#


****************
Another exception has been detected while we were handling last error.
Dumping information about last error:
ERROR REPORT FILE = hs_err_pid22448.log
PC                = 0x4002cb28
SIGNAL            = 11
FUNCTION NAME     = (N/A)
OFFSET            = 0x4002CB28
LIBRARY NAME      = /lib/libpthread.so.0
Please check ERROR REPORT FILE for further information, if there is any.
Good bye.


Comment 7 Joseph Shraibman 2003-10-08 04:07:41 UTC
Also interesting:  On my single athlon laptop:
$ strace date 2>&1 | grep lib
open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
open("/lib/tls/librt.so.1", O_RDONLY)   = 3
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
$ uname -a
Linux jks-laptop 2.4.20-20.9 #1 Mon Aug 18 11:27:43 EDT 2003 i686 athlon i386
GNU/Linux

On my dual athlon server:
]# strace date 2>&1 | grep lib
open("/lib/libpthread.so.0", O_RDONLY)  = 3
open("/lib/librt.so.1", O_RDONLY)       = 3
open("/lib/libc.so.6", O_RDONLY)        = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
]# uname -a
Linux p2.selectacast.net 2.4.20-20.9smp #1 SMP Mon Aug 18 11:18:01 EDT 2003 i686
athlon i386 GNU/Linux


Comment 8 Joseph Shraibman 2003-10-08 22:57:17 UTC
Never mind. It seems that during the upgrade somehow the i386 version of glibc
was installed instead of the i686 version.  So this isn't a glibc problem but a
different problem.

Perhaps a check can be added to the rpms to make sure that at least a warning is
printed if the i386 rpm gets installed on a better system?  Redhat won't run on
a 386 anymore anyway.


Note You need to log in before you can comment on or make changes to this bug.