Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 822971 - deadlock when unregistering event callbacks
Summary: deadlock when unregistering event callbacks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Eric Blake
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-05-18 16:13 UTC by Zeeshan Ali
Modified: 2016-09-20 01:41 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-29 02:00:09 UTC


Attachments (Terms of Use)

Description Zeeshan Ali 2012-05-18 16:13:59 UTC
Description of problem:

I got session libvirtd hung after I tried to managed-save a Windows XP domain. Here is the bt:

Thread 11 (Thread 0x7fcbb4e45700 (LWP 23186)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4e45700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fcbb4644700 (LWP 23187)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4644700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fcbb3e43700 (LWP 23188)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3e43700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7fcbb3642700 (LWP 23189)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3642700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7fcbb2e41700 (LWP 23190)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2e41700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7fcbb2640700 (LWP 23191)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2640700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7fcbb1e3f700 (LWP 23192)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb1e3f700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7fcbb163e700 (LWP 23193)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb163e700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fcbb0e3d700 (LWP 23194)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb0e3d700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fcbb063c700 (LWP 23195)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb063c700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fcbb526a840 (LWP 23185)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000030bfa09f97 in _L_lock_863 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00000030bfa09deb in __pthread_mutex_lock (mutex=0x7fcba8066628) at pthread_mutex_lock.c:65
#3  0x00007fcbb69ce0d0 in virDomainEventStateLock (state=0x7fcba8066610) at conf/domain_event.c:598
#4  virDomainEventStateDeregisterConn (conn=0x7fcba4000ca0, state=0x7fcba8066610) at conf/domain_event.c:1625
#5  0x0000000000464afc in qemudClose (conn=0x7fcba4000ca0) at qemu/qemu_driver.c:951
#6  0x00007fcbb69f4c4b in virReleaseConnect (conn=0x7fcba4000ca0) at datatypes.c:114
#7  0x00007fcbb69f4fa8 in virUnrefConnect (conn=0x7fcba4000ca0) at datatypes.c:149
#8  0x00007fcbb69cc487 in virDomainEventCallbackListPurgeMarked (cbList=0x7fcba8062ba0) at conf/domain_event.c:351
#9  virDomainEventStateFlush (state=0x7fcba8066610) at conf/domain_event.c:1422
#10 virDomainEventTimer (timer=<optimized out>, opaque=0x7fcba8066610) at conf/domain_event.c:637
#11 0x00007fcbb6976348 in virEventPollDispatchTimeouts () at util/event_poll.c:440
#12 virEventPollRunOnce () at util/event_poll.c:633
#13 0x00007fcbb6974c55 in virEventRunDefaultImpl () at util/event.c:247
#14 0x00007fcbb6a5813d in virNetServerRun (srv=0x1b9fbd0) at rpc/virnetserver.c:713
#15 0x0000000000420ddc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1133

Version-Release number of selected component (if applicable): git master (commit f300c194fdfbd519013563dce5d410e32d7c24b8)

How reproducible: sometimes

Steps to Reproduce: virsh managedsave DOMAIN_NAME
  
Actual results: domain is saved but libvirtd hangs

Expected results: domain is saved and libvirt continues to run..

Comment 1 Daniel Berrange 2012-05-21 11:20:16 UTC
As can be seen from the stack trace this is not actually anything todo with managed save. It is a simple deadlock in the event handling code

https://www.redhat.com/archives/libvir-list/2012-May/msg00995.html

Comment 2 Zeeshan Ali 2013-06-24 23:14:52 UTC
AFAICT, this is fixed. Since I filed this, I'll mark this directly as "VERIFIED"


Note You need to log in before you can comment on or make changes to this bug.