Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1684022 - Guest panic if hotplug and unplug memory while guest is booting
Summary: Guest panic if hotplug and unplug memory while guest is booting
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Igor Mammedov
QA Contact: Yumei Huang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-28 09:46 UTC by Yumei Huang
Modified: 2019-04-06 23:22 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Yumei Huang 2019-02-28 09:46:39 UTC
Description of problem:
Hotplug and unplug memory while guest is booting, guest hit kernel panic.

[    8.299728] BUG: unable to handle kernel NULL pointer dereference at 0000000000000610
[    8.304717] PGD 0 P4D 0 
[    8.307151] Oops: 0000 [#1] SMP PTI
[    8.309265] CPU: 3 PID: 7 Comm: kworker/u8:0 Not tainted 4.18.0-74.el8.x86_64 #1
[    8.313662] Hardware name: Red Hat KVM, BIOS 1.12.0-1.module+el8+2706+3c6581b6 04/01/2014
[    8.318545] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[    8.321759] RIP: 0010:__remove_pages+0x33/0x550
[    8.324459] Code: 41 55 41 54 49 89 f4 55 48 89 fd 53 48 89 d3 48 83 ec 60 48 89 0c 24 65 48 8b 04 25 28 00 00 00 48 89 44 24 58 31 c0 48 89 f8 <48> 2b 47 50 48 c1 f8 06 69 c0 a7 37 bd e9 83 f8 04 0f 85 82 03 00
[    8.335295] RSP: 0018:ffffb9db4066bcf0 EFLAGS: 00010246
[    8.338395] RAX: 00000000000005c0 RBX: 0000000000040000 RCX: 0000000000000000
[    8.342570] RDX: 0000000000040000 RSI: 0000000000140000 RDI: 00000000000005c0
[    8.346732] RBP: 00000000000005c0 R08: 0000000100000000 R09: 0000000040000000
[    8.350886] R10: 0000000040000000 R11: 0000000140000000 R12: 0000000000140000
[    8.355036] R13: 0000000000000000 R14: 0000000000140000 R15: 0000000000040000
[    8.359173] FS:  0000000000000000(0000) GS:ffff9bc1fdb00000(0000) knlGS:0000000000000000
[    8.363869] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    8.367221] CR2: 0000000000000610 CR3: 000000001460a000 CR4: 00000000001406e0
[    8.371795] Call Trace:
[    8.373362]  ? memblock_isolate_range+0xc4/0x139
[    8.376078]  ? firmware_map_remove+0x93/0x9d
[    8.378616]  arch_remove_memory+0x7b/0xc0
[    8.381014]  remove_memory+0x96/0xc0
[    8.383164]  acpi_memory_device_remove+0x65/0xf0
[    8.385900]  acpi_bus_trim+0x55/0x90
[    8.388046]  acpi_device_hotplug+0x2ed/0x460
[    8.390578]  acpi_hotplug_work_fn+0x1a/0x30
[    8.393080]  process_one_work+0x1a7/0x360
[    8.395476]  worker_thread+0x30/0x390
[    8.397652]  ? pwq_unbound_release_workfn+0xd0/0xd0
[    8.400517]  kthread+0x112/0x130
[    8.402480]  ? kthread_bind+0x30/0x30
[    8.404681]  ret_from_fork+0x35/0x40
[    8.406828] Modules linked in: xfs libcrc32c dm_multipath sd_mod ata_generic bochs_drm drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul libata crc32_pclmul crc32c_intel drm ghash_clmulni_intel serio_raw virtio_net virtio_scsi net_failover failover sunrpc dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
[    8.430495] CR2: 0000000000000610
[    8.432505] ---[ end trace 40fa1358d48beb9f ]---
[    8.440637] RIP: 0010:__remove_pages+0x33/0x550
[    8.444639] Code: 41 55 41 54 49 89 f4 55 48 89 fd 53 48 89 d3 48 83 ec 60 48 89 0c 24 65 48 8b 04 25 28 00 00 00 48 89 44 24 58 31 c0 48 89 f8 <48> 2b 47 50 48 c1 f8 06 69 c0 a7 37 bd e9 83 f8 04 0f 85 82 03 00
[    8.460901] RSP: 0018:ffffb9db4066bcf0 EFLAGS: 00010246
[    8.464968] RAX: 00000000000005c0 RBX: 0000000000040000 RCX: 0000000000000000
[    8.469142] RDX: 0000000000040000 RSI: 0000000000140000 RDI: 00000000000005c0
[    8.473279] RBP: 00000000000005c0 R08: 0000000100000000 R09: 0000000040000000
[    8.477420] R10: 0000000040000000 R11: 0000000140000000 R12: 0000000000140000
[    8.481560] R13: 0000000000000000 R14: 0000000000140000 R15: 0000000000040000
[    8.489142] FS:  0000000000000000(0000) GS:ffff9bc1fdb00000(0000) knlGS:0000000000000000
[    8.495044] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    8.498015] CR2: 0000000000000610 CR3: 000000001460a000 CR4: 00000000001406e0
[    8.501451] Kernel panic - not syncing: Fatal exception
[    8.504781] Kernel Offset: 0x35c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    8.509497] ---[ end Kernel panic - not syncing: Fatal exception ]---


Version-Release number of selected component (if applicable):
qemu-kvm-3.1.0-18.module+el8+2834+fa8bb6e2
guest kernel: 4.18.0-74.el8.x86_64
host kernel: 4.18.0-68.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. Run following two scripts to repeat doing hotplug and unplug memory during guest boot

# ./test.sh &  ./hotplug.sh

# cat test.sh 
/usr/libexec/qemu-kvm  \
 -name "mouse-vm" \
 -cpu  IvyBridge\
 -sandbox off \
 -machine pc \
 -nodefaults  \
 -vga std \
 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
 -blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2,node-name=my_file \
 -blockdev driver=qcow2,node-name=my,file=my_file \
 -device scsi-hd,drive=my \
 -netdev tap,id=tap0,vhost=on \
 -device virtio-net-pci,mac=9a:8a:8b:8c:8d:8e,id=net0,vectors=4,netdev=tap0,bus=pci.0,addr=0x5 \
 -m 4096,slots=256,maxmem=60G\
 -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
 -vnc :10 \
 -rtc base=utc,clock=host \
 -boot menu=off,strict=off,order=cdn,once=c \
 -enable-kvm  \
 -serial tcp:0:4444,server,nowait \
 -numa node -numa node \
 -monitor  unix:/tmp/monitor3,server,nowait


# cat hotplug.sh 
#!/bin/bash
sleep 10
while true
do
	sleep 1
	echo "object_add memory-backend-ram,id=mem0,size=1G" |nc -U /tmp/monitor3 
	echo "device_add pc-dimm,id=dimm0,memdev=mem0,node=1" |nc -U /tmp/monitor3
	sleep 0.1
	echo "device_del dimm0" |nc -U /tmp/monitor3
	echo "object_del mem0" |nc -U /tmp/monitor3
done

2.
3.

Actual results:
Guest hit kernel panic.

Expected results:
Guest could boot up successfully

Additional info:

Comment 1 Yumei Huang 2019-02-28 09:54:00 UTC
Add guest kernel line here, since added extra parameter "movable_node" as suggested by https://bugzilla.redhat.com/show_bug.cgi?id=1654978#c26.

# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-74.el8.x86_64 root=/dev/mapper/rhel_bootp--73--130--247-root ro console=ttyS0,115200 crashkernel=auto resume=/dev/mapper/rhel_bootp--73--130--247-swap rd.lvm.lv=rhel_bootp-73-130-247/root rd.lvm.lv=rhel_bootp-73-130-247/swap biosdevname=0 net.ifnames=0 rhgb quiet console=tty0 movable_node

Comment 2 Yumei Huang 2019-03-07 06:00:36 UTC
Can reproduce by repeating hotplug/unplug for 256 times after guest boot up, but not always. The chance is 2/8.


Note You need to log in before you can comment on or make changes to this bug.