Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 598851 - qpidd c++ broker crash in qmf::org::apache::qpid::broker::Broker::~Broker() -> __gnu_cxx::__exchange_and_add()
Summary: qpidd c++ broker crash in qmf::org::apache::qpid::broker::Broker::~Broker() -...
Keywords:
Status: CLOSED DUPLICATE of bug 566691
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: Development
Hardware: All
OS: Linux
high
high
Target Milestone: 1.3
: ---
Assignee: Ted Ross
QA Contact: MRG Quality Engineering
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-06-02 08:06 UTC by Frantisek Reznicek
Modified: 2015-11-16 01:12 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-10-13 13:24:35 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Frantisek Reznicek 2010-06-02 08:06:33 UTC
Description of problem:

There was observed crash of the qpid c++ broker (qpidd) while looping through PLAIN authentication test in:

  Thread 1 (process 27462):
  #0  0x006c5761 in __gnu_cxx::__exchange_and_add () from /usr/lib/libstdc++.so.6
  #1  0x007a467f in qmf::org::apache::qpid::broker::Broker::~Broker ()
     from /usr/lib/libqpidbroker.so.2
  #2  0x0099e915 in qpid::management::ManagementAgent::periodicProcessing ()
     from /usr/lib/libqpidbroker.so.2
  #3  0x009a23e5 in qpid::management::ManagementAgent::Periodic::fire ()
     from /usr/lib/libqpidbroker.so.2
  #4  0x00514305 in qpid::sys::TimerTask::fireTask ()
     from /usr/lib/libqpidcommon.so.2


The observed crash was seen only on RHEL4.8 i386 so far.

Version-Release number of selected component (if applicable):
python-qpid-0.7.946106-1.el4
qpid-cpp-client-0.7.946106-2.el4
qpid-cpp-client-devel-0.7.946106-2.el4
qpid-cpp-client-devel-docs-0.7.946106-2.el4
qpid-cpp-client-ssl-0.7.946106-2.el4
qpid-cpp-mrg-debuginfo-0.7.946106-1.el4
qpid-cpp-server-0.7.946106-2.el4
qpid-cpp-server-devel-0.7.946106-2.el4
qpid-cpp-server-ssl-0.7.946106-2.el4
qpid-cpp-server-store-0.7.946106-2.el4
qpid-cpp-server-xml-0.7.946106-2.el4
qpid-dotnet-0.4.738274-2.el4
qpid-java-client-0.7.946106-3.el4
qpid-java-common-0.7.946106-3.el4
qpid-tests-0.7.946106-1.el4
qpid-tools-0.7.946106-4.el4


How reproducible:
Very hard, One occurrence during 11075 sec[s] test .

Steps to Reproduce:
1. run the MRG/Messaging/qpid_ptest_authentication_plain test
   or mimic 
   http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_ptest_authentication_plain/runtest.sh?rev=1.20
2. run with TEST_PARAM_RHTS_TEST_MRG_QPID_PTEST_AUTHENTICATION_PLAIN_LOOPS=100 make rb
3. wait for the crash
  
Actual results:
Broker crashes.

Expected results:
Broker should not crash.

Additional info:

The test (MRG/Messaging/qpid_ptest_authentication_plain) was run in extended mode: TEST_PARAM_RHTS_TEST_MRG_QPID_PTEST_AUTHENTICATION_PLAIN_LOOPS=100 make rb


[15:02:05] ========================TEST finished=======================
[15:02:06] ERROR:mrg_core_files_examine:There are 1 recent core files found (total core count is 1)
[15:02:06] mrg_core_files_examine: first 30 cores from following list will be analyzed/dumped:
./core.27457: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qpidd'
[15:02:08] Core file: ./core.27457 generated by qpidd ----------------------1/1-
./core.27457: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "i386-redhat-linux-gnu"...
  warning: the debug information found in "/usr/lib/debug//usr/sbin/qpidd.debug" does not match "/usr/sbin/qpidd" (CRC mismatch).
  Using host libthread_db library "/lib/tls/libthread_db.so.1".
  Core was generated by `qpidd --log-enable=debug+ --auth yes --data-dir /root/MRG/Messaging/qpid_ptest_'.
  Program terminated with signal 11, Segmentation fault.
  warning: the debug information found in "/usr/lib/debug//usr/lib/libqpidbroker.so.2.0.0.debug" does not match "/usr/lib/libqpidbroker.so.2" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/libqpidcommon.so.2.0.0.debug" does not match "/usr/lib/libqpidcommon.so.2" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/ssl.so.debug" does not match "/usr/lib/qpid/daemon/ssl.so" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/libsslcommon.so.2.0.0.debug" does not match "/usr/lib/libsslcommon.so.2" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/xml.so.debug" does not match "/usr/lib/qpid/daemon/xml.so" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/acl.so.debug" does not match "/usr/lib/qpid/daemon/acl.so" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/msgstore.so.debug" does not match "/usr/lib/qpid/daemon/msgstore.so" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/replicating_listener.so.debug" does not match "/usr/lib/qpid/daemon/replicating_listener.so" (CRC mismatch).
  warning: the debug information found in "/usr/lib/debug//usr/lib/qpid/daemon/replication_exchange.so.debug" does not match "/usr/lib/qpid/daemon/replication_exchange.so" (CRC mismatch).
  #0  0x006c5761 in __gnu_cxx::__exchange_and_add () from /usr/lib/libstdc++.so.6
  (gdb) eax            0xffffffff       -1
  ecx            0x93fba00      155171328
  edx            0xfffffffc     -4
  ebx            0xa07e34       10518068
  esp            0xb7f82928     0xb7f82928
  ebp            0xb7f82928     0xb7f82928
  esi            0x8069640      134649408
  edi            0xfffffff4     -12
  eip            0x6c5761       0x6c5761
  eflags         0x10297        66199
  cs             0x73   115
  ss             0x7b   123
  ds             0x7b   123
  es             0x7b   123
  fs             0x0    0
  gs             0x33   51
  (gdb) There are no memory regions defined.
  (gdb) 16   AT_HWCAP             Machine-dependent CPU capability hints 0x178bfbff
  6    AT_PAGESZ            System page size               4096
  17   AT_CLKTCK            Frequency of times()           100
  3    AT_PHDR              Program headers for program    0x8048034
  4    AT_PHENT             Size of program header entry   32
  5    AT_PHNUM             Number of program headers      8
  7    AT_BASE              Base address of interpreter    0x0
  8    AT_FLAGS             Flags                          0x0
  9    AT_ENTRY             Entry point of program         0x804c550
  11   AT_UID               Real user ID                   0
  12   AT_EUID              Effective user ID              0
  13   AT_GID               Real group ID                  0
  14   AT_EGID              Effective group ID             0
  23   AT_SECURE            Boolean, was exec setuid-like? 0
  15   AT_PLATFORM          String identifying platform    0xbff20acb "i686"
  0    AT_NULL              End of vector                  0x0
  (gdb) Stack level 0, frame at 0xb7f82930:
   eip = 0x6c5761 in __gnu_cxx::__exchange_and_add(int volatile*, int);
      saved eip 0x7a467f
   called by frame at 0xb7f829a0
   Arglist at 0xb7f82928, args:
   Locals at 0xb7f82928, Previous frame's sp is 0xb7f82930
   Saved registers:
    ebp at 0xb7f82928, eip at 0xb7f8292c
  (gdb) From        To          Syms Read   Shared Object Library
  0x0079bc90  0x009adfcc  Yes         /usr/lib/libqpidbroker.so.2
  0x003fa4c0  0x005303d0  Yes         /usr/lib/libqpidcommon.so.2
  0x00c60214  0x00c831fc  Yes         /usr/lib/libboost_program_options.so.1
  0x00c91b8c  0x00c99a70  Yes         /usr/lib/libboost_filesystem.so.1
  0x00c4a020  0x00c4b7b4  Yes         /lib/tls/libuuid.so.1
  0x00a650d0  0x00a6a66c  Yes         /lib/tls/librt.so.1
  0x00a4f060  0x00a5d500  Yes         /usr/lib/libsasl2.so.2
  0x00c43bb0  0x00c448c4  Yes         /lib/libdl.so.2
  0x002343f0  0x0023fecc  Yes         /lib/libresolv.so.2
  0x02c368d0  0x02c3c400  Yes         /lib/libcrypt.so.1
  0x00668c30  0x006d1188  Yes         /usr/lib/libstdc++.so.6
  0x00c21300  0x00c3a5a4  Yes         /lib/tls/libm.so.6
  0x002e47e8  0x002eb12c  Yes         /lib/libgcc_s.so.1
  0x00b00c70  0x00bf6f70  Yes         /lib/tls/libc.so.6
  0x00d502d0  0x00d58ab8  Yes         /lib/tls/libpthread.so.0
  0x00ad27a0  0x00ae4caf  Yes         /lib/ld-linux.so.2
  0x00119640  0x00125208  Yes         /usr/lib/qpid/daemon/ssl.so
  0x00fd6610  0x00ff1688  Yes         /usr/lib/libsslcommon.so.2
  0x02fe68b0  0x030c6f0c  Yes         /usr/lib/libnss3.so
  0x02d79ac0  0x02d9d91c  Yes         /usr/lib/libssl3.so
  0x02d49cd0  0x02d6a7b0  Yes         /usr/lib/libnspr4.so
  0x02da8d30  0x02db373c  Yes         /usr/lib/libnssutil3.so
  0x00129f70  0x0012b84c  Yes         /usr/lib/libplc4.so
  0x0012db88  0x0012e97c  Yes         /usr/lib/libplds4.so
  0x00d3b5b0  0x00d44abc  Yes         /usr/lib/libz.so.1
  0x00137ed0  0x00143848  Yes         /usr/lib/qpid/daemon/xml.so
  0x0772c1a0  0x078dd330  Yes         /usr/lib/libxerces-c.so.28
  0x01102ba0  0x012afd94  Yes         /usr/lib/libxqilla.so.3
  0x001548e0  0x00176c78  Yes         /usr/lib/qpid/daemon/acl.so
  0x00d8b100  0x00e4e4d0  Yes         /usr/lib/qpid/daemon/msgstore.so
  0x00e8d900  0x00f3e390  Yes         /usr/lib/tls/i686/libdb_cxx-4.2.so
  0x001803ec  0x001806a3  Yes         /usr/lib/libaio.so.1
  0x00fb6da0  0x00fc2364  Yes         /usr/lib/qpid/daemon/replicating_listener.so
  0x00186f40  0x0018e1d4  Yes         /usr/lib/qpid/daemon/replication_exchange.so
  (gdb)   2 process 27457  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  * 1 process 27462  0x006c5761 in __gnu_cxx::__exchange_and_add ()
     from /usr/lib/libstdc++.so.6
  Thread 2 (process 27457):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00ba909b in __write_nocancel () from /lib/tls/libc.so.6
  #2  0x00b4a74f in _IO_new_file_write () from /lib/tls/libc.so.6
  #3  0x00b4a99c in _IO_new_file_xsputn () from /lib/tls/libc.so.6
  #4  0x00b261e9 in buffered_vfprintf () from /lib/tls/libc.so.6
  #5  0x00b2638b in vfprintf () from /lib/tls/libc.so.6
  #6  0x00b2ed2f in fprintf () from /lib/tls/libc.so.6
  #7  0x00b3cec4 in perror () from /lib/tls/libc.so.6
  #8  0x0097a98a in qpid::management::ManagementAgent::~ManagementAgent$delete ()
     from /usr/lib/libqpidbroker.so.2
  #9  0x00841689 in qpid::broker::Broker::~Broker$delete ()
     from /usr/lib/libqpidbroker.so.2
  #10 0x0083cae7 in qpid::RefCounted::released ()
     from /usr/lib/libqpidbroker.so.2
  #11 0x08057a4b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<char*> ()
  #12 0x08050aea in ?? ()
  #13 0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6
  #14 0x0804c571 in ?? ()
  Thread 1 (process 27462):
  #0  0x006c5761 in __gnu_cxx::__exchange_and_add () from /usr/lib/libstdc++.so.6
  #1  0x007a467f in qmf::org::apache::qpid::broker::Broker::~Broker ()
     from /usr/lib/libqpidbroker.so.2
  #2  0x0099e915 in qpid::management::ManagementAgent::periodicProcessing ()
     from /usr/lib/libqpidbroker.so.2
  #3  0x009a23e5 in qpid::management::ManagementAgent::Periodic::fire ()
     from /usr/lib/libqpidbroker.so.2
  #4  0x00514305 in qpid::sys::TimerTask::fireTask ()
     from /usr/lib/libqpidcommon.so.2
  #5  0x005143b3 in qpid::sys::Timer::fire () from /usr/lib/libqpidcommon.so.2
  #6  0x00518c26 in qpid::sys::Timer::run () from /usr/lib/libqpidcommon.so.2
  #7  0x004157a1 in qpid::sys::(anonymous namespace)::runRunnable ()
     from /usr/lib/libqpidcommon.so.2
  #8  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #9  0x00bb7fae in clone () from /lib/tls/libc.so.6

[15:02:09] mrg_rhts_report_result:Following core files left in directory:
core.27457
[15:02:09] ====================================================================
[15:02:09] ====================================================================
[15:02:09] Test Summary:    FAIL   #TESTS:16000  #FAILS:2001
[15:02:09] Test Name:       /distribution/MRG/Messaging/qpid_ptest_authentication_plain
[15:02:09] Test Log:        /root/MRG/Messaging/qpid_ptest_authentication_plain/qpid_ptest_authentication_plain.log
[15:02:09] Test Timestamps: 2010-06-01 11:58:04 -> 2010-06-01 15:02:09  duration: 11075 sec[s]
[15:02:09] Test Space:      137020 KB (left)
[15:02:09] Test MRG pkgs:   mrg-rhel4-candidate on RHEL 4.8 i386
[15:02:09] ====================================================================
[15:02:09] Test-case[s] Summary: (1 found)
[15:02:09] V.MRG.M.0005: FAIL  #tests:16000  #fails:2000  desc.:"qpidd broker authentication SASL Plain"
[15:02:09] ====================================================================
[15:02:09] ====================================================================
/distribution/MRG/Messaging/qpid_ptest_authentication_plain result: FAIL
   metric: 2000
   Log: /root/MRG/Messaging/qpid_ptest_authentication_plain/qpid_ptest_authentication_plain.log
[15:02:11] mrg_exit:Exiting with ecode:2000 [from ./runtest.sh:164]

Comment 1 Gordon Sim 2010-06-07 16:38:29 UTC
I believe this is likely a duplicate of bug 566691, where the timer  tries to execute the management agents periodic task, accessing deleted objects in the process. Fix checked in on trunk (r952307) and in release repo
(http://mrg1.lab.bos.redhat.com/git/?p=qpid.git;a=commitdiff;h=79804fa332a5c1b7a2ca7da5cbee843f91f7fa3b).

Comment 2 Ted Ross 2010-10-13 13:24:35 UTC

*** This bug has been marked as a duplicate of bug 566691 ***


Note You need to log in before you can comment on or make changes to this bug.