Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 591076 - migration will cause libvirtd crash
Summary: migration will cause libvirtd crash
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: All
OS: Linux
low
high
Target Milestone: rc
: ---
Assignee: Daniel Berrange
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-05-11 11:37 UTC by Johnny Liu
Modified: 2010-07-02 19:33 UTC (History)
6 users (show)

Fixed In Version: libvirt-0.8.1-5.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-07-02 19:33:19 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Johnny Liu 2010-05-11 11:37:59 UTC
Description of problem:
migration will cause libvirtd crash

Version-Release number of selected component (if applicable):
libvirt-0.8.1-2.el6.x86_64
qemu-kvm-0.12.1.2-2.51.el6.x86_64
kernel-2.6.32-24.el6.x86_64
# rpm -qa|grep selinux
selinux-policy-targeted-3.7.19-12.el6.noarch
selinux-policy-3.7.19-12.el6.noarch
libselinux-2.0.90-3.el6.x86_64
libselinux-python-2.0.90-3.el6.x86_64
libselinux-utils-2.0.90-3.el6.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Install RHEL6.0-20100508.n.0 on two machines.

2. The selinux is in enforcing mode on the two machines
# getenforce 
Enforcing

3. Mount a nfs directory including vm image to local mount point on both source and target hosts.
# mount -o context="system_u:object_r:svirt_image_t:s0" 10.66.90.113:/vol/xenimage /mnt

4. Define domain on source host
# virsh dumpxml demo
<domain type='kvm'>
  <name>demo</name>
  <uuid>a0e61e7b-3b15-dcc0-e487-7c787bd960a1</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu'/>
      <source file='/mnt/jialiu/demo.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:5e:49:1a'/>
      <source network='default'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
</domain>

5. Start domain on source host
# virsh start demo
Domain demo started
# virsh list --all
 Id Name                 State
----------------------------------
  1 demo                 running


6. Live Migrate domain from source host to target host
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
root@10.66.70.86's password: 
error: internal error canonical hostname pointed to localhost, but this is not allowed
(10.66.70.86 is target host IP)

7. According to BZ#580827, there is a workaround, so I change /etch/hosts as following on source host and target host
Target Host:
# cat /etc/hosts
#127.0.0.1	dhcp-66-70-86.nay.redhat.com	localhost.localdomain localhost
127.0.0.1	localhost localhost.localdomain
10.66.70.86	dhcp-66-70-86.nay.redhat.com
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Source Host:
# cat /etc/hosts
#127.0.0.1	dhcp-66-70-178.nay.redhat.com	localhost.localdomain localhost
127.0.0.1       localhost localhost.localdomain
10.66.70.86	dhcp-66-70-86.nay.redhat.com
:1         localhost localhost.localdomain localhost6 localhost6.localdomain6

8. Migrate again.
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
root@10.66.70.86's password: 
error: operation failed: Migration unexpectedly failed
# virsh migrate --live demo qemu+ssh://10.66.70.86/system
error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: Connection refused
error: failed to connect to the hypervisor
# virsh list --all
error: unable to connect to '/var/run/libvirt/libvirt-sock', libvirtd may need to be started: Connection refused
error: failed to connect to the hypervisor
# service libvirtd status
libvirtd dead but pid file exists

9. Try to restart libvirtd, but still failed.
# service libvirtd restart
Stopping libvirtd daemon:                                  [FAILED]
Starting libvirtd daemon:                                  [  OK  ]
# service libvirtd status
libvirtd dead but pid file exists

10. Check /var/log/message, get the following error:
...
May 11 06:23:52 dhcp-66-70-178 kernel: libvirtd[2695]: segfault at 0 ip 000000395d6800f2 sp 00007fc8b6cdf5a8 error 4 in libc-2.12.so[395d600000+175000]
...
  
Actual results:
Migration failed, and it cause libvirtd crash, even can not be restart.

Expected results:
Migration should be done successfully.

Additional info:

Comment 2 RHEL Product and Program Management 2010-05-11 13:11:55 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 4 Dave Allan 2010-05-19 03:43:50 UTC
libvirt-0.8.1-5.el6 has been built in RHEL-6-candidate with the fix

Dave

Comment 7 zhanghaiyan 2010-06-13 06:00:46 UTC
Verified this bug PASS with libvirt-0.8.1-8.el6.x86_64, no crash is seen anymore
- kernel-2.6.32-33.el6.x86_64
- qemu-img-0.12.1.2-2.73.el6.x86_64
- qemu-kvm-0.12.1.2-2.73.el6.x86_64

Comment 8 releng-rhel@redhat.com 2010-07-02 19:33:19 UTC
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.