Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 589287 - broker deadlock between InactivityFireEvent::fire() and MessageStoreImpl::dequeue()
Summary: broker deadlock between InactivityFireEvent::fire() and MessageStoreImpl::deq...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: Development
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: 1.3
: ---
Assignee: Kim van der Riet
QA Contact: MRG Quality Engineering
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-05-05 18:32 UTC by Gordon Sim
Modified: 2010-05-17 07:42 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-05-17 07:42:37 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Gordon Sim 2010-05-05 18:32:16 UTC
Description of problem:

Deadlock in broker.

Version-Release number of selected component (if applicable):

qpid-cpp-server-0.7.935473-1.el5

How reproducible:

Not sure

Steps to Reproduce:
1. run durable perftest with severl pubs/subs
  
Actual results:

Broker hangs:

Thread 5 (Thread 0x410c9940 (LWP 19752)):
#0  0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x00000039d24074d8 in qpid::sys::Timer::run ()
#2  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x4200b940 (LWP 19753)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b85ee95b65a in mrg::journal::slock::slock ()
#4  0x00002b85ee96742c in mrg::journal::jcntl::flush ()
#5  0x00002b85ee918b0b in mrg::msgstore::JournalImpl::flush ()
#6  0x00002b85ee918dc6 in mrg::msgstore::JournalImpl::flushFire ()
#7  0x00002b85ee918e4c in mrg::msgstore::InactivityFireEvent::fire ()
#8  0x00000039d2407123 in qpid::sys::Timer::run ()
#9  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#10 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#11 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x42a0c940 (LWP 19754)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa40b16b in pthread_cond_signal@@GLIBC_2.3.2 ()
#2  0x00000039d29581fd in qpid::broker::IncompleteMessageList::enqueueComplete
#3  0x00000039d29596df in boost::function1<void, boost::intrusive_ptr<qpid::broker::Message> const&, std::allocator<void> >::operator() ()
#4  0x00000039d2975425 in qpid::broker::Message::allEnqueuesComplete ()
#5  0x00000039d297c339 in qpid::broker::PersistableMessage::enqueueComplete ()
#6  0x00002b85ee916d86 in mrg::msgstore::JournalImpl::wr_aio_cb ()
#7  0x00002b85ee9894a7 in mrg::journal::wmgr::get_events ()
#8  0x00002b85ee98bd7f in mrg::journal::wmgr::write_flush ()
#9  0x00002b85ee98c14d in mrg::journal::wmgr::flush_check ()
#10 0x00002b85ee98d0bc in mrg::journal::wmgr::dequeue ()
#11 0x00002b85ee968064 in mrg::journal::jcntl::dequeue_data_record ()
#12 0x00002b85ee91c973 in mrg::msgstore::JournalImpl::dequeue_data_record ()
#13 0x00002b85ee9277ea in mrg::msgstore::MessageStoreImpl::async_dequeue ()
#14 0x00002b85ee93761f in mrg::msgstore::MessageStoreImpl::dequeue ()
#15 0x00000039d29794eb in qpid::broker::MessageStoreModule::dequeue ()
#16 0x00000039d2983db7 in qpid::broker::Queue::dequeue ()
#17 0x00000039d29b2674 in qpid::broker::SemanticState::ConsumerImpl::deliver ()
#18 0x00000039d2985eac in qpid::broker::Queue::dispatch ()
#19 0x00000039d29b29c4 in qpid::broker::SemanticState::ConsumerImpl::doOutput
#20 0x00000039d23f9a87 in qpid::sys::AggregateOutput::doOutput ()
#21 0x00000039d29201e1 in qpid::broker::Connection::doOutput ()
#22 0x00000039d28fe616 in qpid::amqp_0_10::Connection::canEncode ()
#23 0x00000039d29b0bda in qpid::broker::SecureConnection::canEncode ()
#24 0x00000039d23fd7a5 in qpid::sys::AsynchIOHandler::idle ()
#25 0x00000039d231e14f in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2
#26 0x00000039d231bb2f in qpid::sys::posix::AsynchIO::writeable ()
#27 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#28 0x00000039d23ffde7 in qpid::sys::DispatchHandle::processEvent ()
#29 0x00000039d2329fff in qpid::sys::Poller::run ()
#30 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#31 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#32 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4480f940 (LWP 19757)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b85ee95b65a in mrg::journal::slock::slock ()
#4  0x00002b85ee968052 in mrg::journal::jcntl::dequeue_data_record ()
#5  0x00002b85ee91c973 in mrg::msgstore::JournalImpl::dequeue_data_record ()
#6  0x00002b85ee9277ea in mrg::msgstore::MessageStoreImpl::async_dequeue ()
#7  0x00002b85ee93761f in mrg::msgstore::MessageStoreImpl::dequeue ()
#8  0x00000039d29794eb in qpid::broker::MessageStoreModule::dequeue ()
#9  0x00000039d2983db7 in qpid::broker::Queue::dequeue ()
#10 0x00000039d298418c in qpid::broker::Queue::popAndDequeue ()
#11 0x00000039d29846ba in qpid::broker::Queue::purge ()
#12 0x00000039d29c884f in qpid::broker::SessionAdapter::QueueHandlerImpl::purge
#13 0x00000039d23467b2 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /usr/lib64/libqpidcommon.so.2
#14 0x00000039d234cf17 in qpid::framing::AMQP_ServerOperations::Invoker::visit
#15 0x00000039d29e16e3 in qpid::framing::invoke<qpid::broker::SessionAdapter>
#16 0x00000039d29dff7d in qpid::broker::SessionState::handleCommand ()
#17 0x00000039d29e0ef9 in qpid::broker::SessionState::handleIn ()
#18 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ()
#19 0x00000039d291dee2 in qpid::broker::Connection::received ()
#20 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode ()
#21 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff ()
#22 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#23 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable ()
#24 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#25 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent ()
#26 0x00000039d2329fff in qpid::sys::Poller::run ()
#27 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#28 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#29 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b85ec35f830 (LWP 19751)):
#0  0x00000038fa407955 in pthread_join () from /lib64/libpthread.so.0
#1  0x00000039d23203cd in qpid::sys::Thread::join ()
#2  0x00000039d290a311 in qpid::broker::Broker::run ()
#3  0x0000000000406ae6 in ?? ()
#4  0x00000000004055af in __cxa_pure_virtual ()
#5  0x00000038f9c1d994 in __libc_start_main () from /lib64/libc.so.6
#6  0x0000000000405019 in __cxa_pure_virtual ()
#7  0x00007fffabd2a678 in ?? ()
#8  0x0000000000000000 in ?? ()

Comment 1 Gordon Sim 2010-05-06 17:19:05 UTC
Another occurence, it appears relatively easy (I got it well within 50 loops) to trigger this via:

   while perftest --qt 2 --nsubs 4 --npubs 2 --sub-ack 10 --durable true --count 50000 --summary; do true; done

Thread 7 (Thread 0x40d70940 (LWP 6789)):
#0  0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x00000039d24074d8 in qpid::sys::Timer::run ()
#2  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x41771940 (LWP 6790)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b761173865a in mrg::journal::slock::slock ()
#4  0x00002b761174442c in mrg::journal::jcntl::flush ()
#5  0x00002b76116f5b0b in mrg::msgstore::JournalImpl::flush ()
#6  0x00002b76116f5dc6 in mrg::msgstore::JournalImpl::flushFire ()
#7  0x00002b76116f5e4c in mrg::msgstore::InactivityFireEvent::fire ()
#8  0x00000039d2407123 in qpid::sys::Timer::run ()
#9  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#10 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#11 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x429a5940 (LWP 6791)):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000039d23295af in qpid::sys::Poller::wait ()
#2  0x00000039d2329fd2 in qpid::sys::Poller::run ()
#3  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x433a6940 (LWP 6792)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b761173865a in mrg::journal::slock::slock ()
#4  0x00002b7611745052 in mrg::journal::jcntl::dequeue_data_record ()
#5  0x00002b76116f9973 in mrg::msgstore::JournalImpl::dequeue_data_record ()
#6  0x00002b76117047ea in mrg::msgstore::MessageStoreImpl::async_dequeue ()
#7  0x00002b761171461f in mrg::msgstore::MessageStoreImpl::dequeue ()
#8  0x00000039d29794eb in qpid::broker::MessageStoreModule::dequeue ()
#9  0x00000039d2983db7 in qpid::broker::Queue::dequeue ()
#10 0x00000039d292ed34 in qpid::broker::DeliveryRecord::accept ()
#11 0x00000039d29c1853 in std::__find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > ()
#12 0x00000039d29c1ed9 in std::remove_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > ()
#13 0x00000039d29b70e3 in qpid::broker::SemanticState::accepted ()
#14 0x00000039d23470c2 in qpid::framing::AMQP_ServerOperations::MessageHandler::Invoker::visit () from /usr/lib64/libqpidcommon.so.2
#15 0x00000039d234abd7 in qpid::framing::AMQP_ServerOperations::Invoker::visit
#16 0x00000039d29e16e3 in qpid::framing::invoke<qpid::broker::SessionAdapter>
#17 0x00000039d29dff7d in qpid::broker::SessionState::handleCommand ()
#18 0x00000039d29e0ef9 in qpid::broker::SessionState::handleIn ()
#19 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ()
#20 0x00000039d291dee2 in qpid::broker::Connection::received ()
#21 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode ()
#22 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff ()
#23 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#24 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable ()
#25 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#26 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent ()
#27 0x00000039d2329fff in qpid::sys::Poller::run ()
#28 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#29 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#30 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x43da7940 (LWP 6793)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa40b16b in pthread_cond_signal@@GLIBC_2.3.2 ()
#2  0x00000039d29581fd in qpid::broker::IncompleteMessageList::enqueueComplete
#3  0x00000039d29596df in boost::function1<void, boost::intrusive_ptr<qpid::broker::Message> const&, std::allocator<void> >::operator() ()
#4  0x00000039d2975425 in qpid::broker::Message::allEnqueuesComplete ()
#5  0x00000039d297c339 in qpid::broker::PersistableMessage::enqueueComplete ()
#6  0x00002b76116f3d86 in mrg::msgstore::JournalImpl::wr_aio_cb ()
#7  0x00002b76117664a7 in mrg::journal::wmgr::get_events ()
#8  0x00002b7611768d7f in mrg::journal::wmgr::write_flush ()
#9  0x00002b761176914d in mrg::journal::wmgr::flush_check ()
#10 0x00002b761176a921 in mrg::journal::wmgr::enqueue ()
#11 0x00002b76117453be in mrg::journal::jcntl::enqueue_data_record ()
#12 0x00002b76116fa283 in mrg::msgstore::JournalImpl::enqueue_data_record ()
#13 0x00002b76117051a7 in mrg::msgstore::MessageStoreImpl::store ()
#14 0x00002b761171301b in mrg::msgstore::MessageStoreImpl::enqueue ()
#15 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue ()
#16 0x00000039d298860c in qpid::broker::Queue::enqueue ()
#17 0x00000039d2989dc6 in qpid::broker::Queue::deliver ()
#18 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo ()
#19 0x00000039d2944065 in qpid::broker::Exchange::doRoute ()
#20 0x00000039d2930ed3 in qpid::broker::DirectExchange::route ()
#21 0x00000039d29babf3 in qpid::broker::SemanticState::route ()
#22 0x00000039d29bbb4d in qpid::broker::SemanticState::handle ()
#23 0x00000039d29e090e in qpid::broker::SessionState::handleContent ()
#24 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn ()
#25 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ()
#26 0x00000039d291dee2 in qpid::broker::Connection::received ()
#27 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode ()
#28 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff ()
#29 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#30 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable ()
#31 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#32 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent ()
#33 0x00000039d2329fff in qpid::sys::Poller::run ()
#34 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#35 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#36 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x447a8940 (LWP 6794)):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b761173865a in mrg::journal::slock::slock ()
#4  0x00002b7611745052 in mrg::journal::jcntl::dequeue_data_record ()
#5  0x00002b76116f9973 in mrg::msgstore::JournalImpl::dequeue_data_record ()
#6  0x00002b76117047ea in mrg::msgstore::MessageStoreImpl::async_dequeue ()
#7  0x00002b761171461f in mrg::msgstore::MessageStoreImpl::dequeue ()
#8  0x00000039d29794eb in qpid::broker::MessageStoreModule::dequeue ()
#9  0x00000039d2983db7 in qpid::broker::Queue::dequeue ()
#10 0x00000039d292ed34 in qpid::broker::DeliveryRecord::accept ()
#11 0x00000039d29c1853 in std::__find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > ()
#12 0x00000039d29c1ed9 in std::remove_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > ()
#13 0x00000039d29b70e3 in qpid::broker::SemanticState::accepted ()
#14 0x00000039d23470c2 in qpid::framing::AMQP_ServerOperations::MessageHandler::Invoker::visit () from /usr/lib64/libqpidcommon.so.2
#15 0x00000039d234abd7 in qpid::framing::AMQP_ServerOperations::Invoker::visit
#16 0x00000039d29e16e3 in qpid::framing::invoke<qpid::broker::SessionAdapter>
#17 0x00000039d29dff7d in qpid::broker::SessionState::handleCommand ()
#18 0x00000039d29e0ef9 in qpid::broker::SessionState::handleIn ()
#19 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ()
#20 0x00000039d291dee2 in qpid::broker::Connection::received ()
#21 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode ()
#22 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff ()
#23 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#24 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable ()
#25 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#26 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent ()
#27 0x00000039d2329fff in qpid::sys::Poller::run ()
#28 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#29 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#30 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b760f13c830 (LWP 6788)):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000039d23295af in qpid::sys::Poller::wait ()
#2  0x00000039d2329fd2 in qpid::sys::Poller::run ()
#3  0x00000039d290a2f2 in qpid::broker::Broker::run ()
#4  0x0000000000406ae6 in ?? ()
#5  0x00000000004055af in __cxa_pure_virtual ()
#6  0x00000038f9c1d994 in __libc_start_main () from /lib64/libc.so.6
#7  0x0000000000405019 in __cxa_pure_virtual ()
#8  0x00007fffb4671038 in ?? ()
#9  0x0000000000000000 in ?? ()
[gordon@mrg15 ~]$

Comment 2 Kim van der Riet 2010-05-06 18:20:43 UTC
In both these cases, it seems that something is holding the lock in qpid::broker::IncompleteMessageList::enqueueComplete() which is blocking the thread. I cannot find another thread that is doing so, however.

In the first case, thread 3 is holding the store lock, but is blocked in the broker callback in IncompleteMessageList::enqueueComplete(). Threads 2 and 4 (which contians the flush call from the timer) are simply waiting for this thread to complete and release the store lock.

The second case is similar. Thread 3 is stuck in the broker callback (aparrently with no thread holding the lock), and threads 2, 4 and 6 (the timer flush call) are waiting for thread 3 to return.

I cannot see what is holding the lock in IncompleteMessageList::enqueueComplete() in either of these cases. None of the other threads have passed through the IncompleteMessageList class, and so cannot be holding the lock.

Comment 3 Gordon Sim 2010-05-17 07:42:37 UTC
I believe this may have been caused by the heap corruption fixed in r944329. Certainly I can't reproduce it now, so closing.


Note You need to log in before you can comment on or make changes to this bug.