Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361035 - [libvirt][xen-4.7]Guest's kdump on Xen hypervisor isn't triggered when managing guest with virsh.
Summary: [libvirt][xen-4.7]Guest's kdump on Xen hypervisor isn't triggered when managi...
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-28 08:29 UTC by Lin Liu
Modified: 2018-07-18 14:58 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description Lin Liu 2016-07-28 08:29:07 UTC
Description of problem:
Guest's kdump on Xen-4.7.0 hypervisor is supported now, but if using virsh command to create a xen guest and trigger the crash, kdump isn't worked, the guest hangs and virsh list still displays the guest is running, guest doesn't reboot and no crash file after rebooting the guest manually.

Version-Release number of selected component (if applicable):
Host Fedora Rawhide with latest libvirt, xen and kernel:
libvirt-2.0.0-2.fc25.x86_64
xen-4.7.0-1.fc25.x86_64
4.7.0-0.rc7.git4.1.fc25.x86_64

Guest RHEL7.3 with kernel:
kernel-3.10.0-470.el7.x86_64

How reproducible: Always

Steps to Reproduce:
1. Start a RHEL-7.3 pre-installed guest image on Fedora Rawhide xen-4.7 host using command "virsh create xen-hvm-guest.xml" with below configuration:
[root@dhcp-9-56 ~]# cat xen-hvm-guest.xml
<domain type='xen'>
  <name>xen-hvm-guest</name>
  <uuid>46d24fd1-9299-4701-98d9-cb82797754d5</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='xenfv'>hvm</type>
    <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='variable' adjustment='0' basis='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <disk type='file' device='disk'>
      <driver name='file'/>
      <source file='/home/RHEL-Server-7.3-64-hvm.raw'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='e8:35:30:4c:4b:00'/>
      <source bridge='xenbr0'/>
      <script path='vif-bridge'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='4096' heads='1' primary='yes'/>
    </video>
  </devices>
</domain>


2. Add kernel parameter `crashkernel=300M` to the guest's `/etc/default/grub` and reboot the guest.
3. Start kdump service with `service kdump start` and confirm kdump service is running:
   # service kdump status
4. Trigger guest crash and see if kdump works:
   # echo "c" > /proc/sysrq-trigger


Actual results:
The guest hangs with output, and check the guest status with virsh list, the guest is running status. Check the guest status with xl list, it status is "---sS-".

[root@dhcp-9-110 ~]# echo "c" > /proc/sysrq-trigger
[   38.323083] SysRq : Trigger a crash
[   38.331067] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   38.332029] IP: [<ffffffff813e2bd6>] sysrq_handle_crash+0x16/0x20
[   38.332029] PGD 79fe5067 PUD 79fe4067 PMD 0
[   38.332029] Oops: 0002 [#1] SMP
[   38.332029] Modules linked in: ipt_REJECT nf_reject_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter intel_powerclamp intel_rapl iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev xen_kbdfront i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm xen_blkfront xen_netfront ata_piix crc32c_intel libata i2c_core serio_raw floppy fjes dm_mirror dm_region_hash dm_log dm_mod
[   38.332029] CPU: 1 PID: 2333 Comm: bash Not tainted 3.10.0-461.el7.x86_64 #1
[   38.332029] Hardware name: Xen HVM domU, BIOS 4.7.0 07/22/2016
[   38.332029] task: ffff8800367cedd0 ti: ffff8800788f0000 task.ti: ffff8800788f0000
[   38.332029] RIP: 0010:[<ffffffff813e2bd6>]  [<ffffffff813e2bd6>] sysrq_handle_crash+0x16/0x20
[   38.332029] RSP: 0018:ffff8800788f3e88  EFLAGS: 00010246
[   38.332029] RAX: 000000000000000f RBX: ffffffff81a73f80 RCX: 0000000000000000
[   38.332029] RDX: 0000000000000000 RSI: ffff88007ce4f848 RDI: 0000000000000063
[   38.332029] RBP: ffff8800788f3e88 R08: 0000000000000086 R09: 0000000000000230
[   38.332029] R10: 000000000000022f R11: 0000000000000003 R12: 0000000000000063
[   38.332029] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000000
[   38.332029] FS:  00007f2012fbf740(0000) GS:ffff88007ce40000(0000) knlGS:0000000000000000
[   38.332029] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   38.332029] CR2: 00007fb6ba2bf000 CR3: 0000000077bbc000 CR4: 00000000001406e0
[   38.332029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   38.332029] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   38.332029] Stack:
[   38.332029]  ffff8800788f3eb8 ffffffff813e33f7 0000000000000002 00007f2012fbd000
[   38.332029]  ffff8800788f3f48 0000000000000002 ffff8800788f3ed0 ffffffff813e386f
[   38.332029]  ffff8800783d5840 ffff8800788f3ef0 ffffffff812646bd 0000000000000002
[   38.332029] Call Trace:
[   38.332029]  [<ffffffff813e33f7>] __handle_sysrq+0x107/0x170
[   38.332029]  [<ffffffff813e386f>] write_sysrq_trigger+0x2f/0x40
[   38.332029]  [<ffffffff812646bd>] proc_reg_write+0x3d/0x80
[   38.332029]  [<ffffffff811f7abd>] vfs_write+0xbd/0x1e0
[   38.332029]  [<ffffffff811f855f>] SyS_write+0x7f/0xe0
[   38.332029]  [<ffffffff8168f349>] system_call_fastpath+0x16/0x1b
[   38.332029] Code: eb 9b 45 01 f4 45 39 65 34 75 e5 4c 89 ef e8 e2 f7 ff ff eb db 0f 1f 44 00 00 55 48 89 e5 c7 05 01 0d 60 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 c7 05 7e
[   38.332029] RIP  [<ffffffff813e2bd6>] sysrq_handle_crash+0x16/0x20
[   38.332029]  RSP <ffff8800788f3e88>
[   38.332029] CR2: 0000000000000000


Expected results:
Guest should reboot into capture kernel and there's a new dump file in guest `/var/crash` after rebooting.

Additional info:
Both Xen-4.7 host and RHEL7.3 guest(after kernel-3.10.0-351.el7) support the kdump feature, and if using xl command to create guest of the same RHEL7.3 image with configure file converted from xml file of step1, the kdump can be triggered, guest reboot and crash file is created. please refer to bug https://bugzilla.redhat.com/show_bug.cgi?id=1007328.

Comment 1 Fedora End Of Life 2017-02-28 10:01:10 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 2 Cole Robinson 2017-05-03 20:51:06 UTC
Are you still seeing this with latest f26/rawhide packages?

Comment 3 Xiao, Liang 2017-05-09 07:59:21 UTC
In my testing on Fedora26 alpha release, I still can see this problem.

Comment 4 Cole Robinson 2017-05-09 16:35:58 UTC
Can you get this working without libvirt in the mix? That should tell us if it's libvirt's fault or xen in general

Comment 5 Lin Liu 2017-06-13 02:51:46 UTC
Yes, it work's OK with xl command without libvirt, 
Both Xen-4.7 host and RHEL7.3 guest(after kernel-3.10.0-351.el7) support the kdump feature, and if using xl command to create guest of the same RHEL7.3 image with configure file converted from xml file of step1, the kdump can be triggered, guest reboot and crash file is created. Please refer to bug https://bugzilla.redhat.com/show_bug.cgi?id=1007328.

Comment 6 Cole Robinson 2017-06-26 18:27:41 UTC
Okay since this has persisted across releases and in fedora we are largely kvm focused, moving to the upstream tracker and CCing libvirt xen maintainer


Note You need to log in before you can comment on or make changes to this bug.