Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1515665 - Call trace happened when stress was running inside guest
Summary: Call trace happened when stress was running inside guest
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: ppc64le
OS: Linux
medium
medium
Target Milestone: rc
: 7.5
Assignee: David Gibson
QA Contact: Min Deng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-21 08:16 UTC by Min Deng
Modified: 2018-01-03 14:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-20 03:34:44 UTC


Attachments (Terms of Use)
calltracelogfromguest (deleted)
2017-11-21 08:17 UTC, Min Deng
no flags Details
stresstool (deleted)
2017-11-21 08:17 UTC, Min Deng
no flags Details

Description Min Deng 2017-11-21 08:16:17 UTC
Description of problem:
Call trace happened when stress was running inside guest

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.10.0-6.el7.ppc64le
kernel-4.14.0-2.el7a.ppc64le
SLOF-20170724-2.git89f519f.el7.noarch

How reproducible:
1/1
Steps to Reproduce:

#mount -t hugetlbfs none /mnt/kvm_hugepage
#echo 40960 > /proc/sys/vm/nr_hugepages
#cat /proc/meminfo | grep -i hugepage

1.boot up guest with the following cli,
  /usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pseries -nodefaults -vga std -chardev socket,id=serial_id_serial0,path=/tmp/S,servet -device spapr-vty,reg=0x30000000,chardev=serial_id_serial0 -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=Pegas-Server-7.4-ppc64le-virtio-scsi.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,vhost=on -device virtio-net-pci,netdev=net0,id=nic0,mac=52:54:00:13:17:51,bus=pci.0,addr=0x1e -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :11 -rtc base=utc,clock=host -enable-kvm -device usb-kbd,id=input0 -device usb-mouse,id=input1 -device usb-tablet,id=input2 -monitor stdio -smp 4 -m 80G,slots=256,maxmem=1T -mem-path /mnt/kvm_hugepage -qmp tcp:0:4444,server,nowait -monitor unix:/tmp/monitor3,server,nowait
2.run stress
  stress --vm 160 --vm-bytes 256M --timeout 30s
3.

Actual results:
call trace,please see attachment since it was very long.


Expected results:
There is no call trace issue.

Additional info:

Comment 1 Min Deng 2017-11-21 08:17:12 UTC
Created attachment 1356445 [details]
calltracelogfromguest

call trace log

Comment 2 Min Deng 2017-11-21 08:17:48 UTC
Created attachment 1356446 [details]
stresstool

Comment 3 Min Deng 2017-11-21 08:18:40 UTC
I will update x86 results as soon as get it,thanks.

Comment 4 Min Deng 2017-11-21 08:19:11 UTC
Host's memory
[root@c155f3-u23 mdeng]# free -m
              total        used        free      shared  buff/cache   available
Mem:         126675       98808       26529          15        1337       26065
Swap:          4095         412        3683

Comment 5 Min Deng 2017-11-22 07:01:38 UTC
Tried it on x86 platform but it wasn't reproduced.So mark it as ppc only

Comment 6 Qunfang Zhang 2017-11-29 09:16:06 UTC
(In reply to Min Deng from comment #5)
> Tried it on x86 platform but it wasn't reproduced.So mark it as ppc only

Min,

It might be good to list the x86 tested version too.  For both kernel and qemu-kvm-rhev version. Especially whether we are testing a kernel-3.10 or kernel-alt-4.11.

Thanks!

Comment 7 Min Deng 2017-11-30 07:02:30 UTC
QE reported the bug on kernel-4.14.0-2.el7a.ppc64le and also tested it on x86 platform and the build info is following as below,any other issues please let me know,thanks a lot.
kernel-3.10.0-799.el7.x86_64
qemu-kvm-rhev-2.10.0-9.el7.x86_64

Comment 8 David Gibson 2017-12-13 04:00:29 UTC
I'm assuming those call traces are from the guest not the host?

Did the x86 VM have the same available RAM as on POWER?

They look like they're just caused by high memory pressure, which is expected.  I think the reason they don't appear on x86 is that PPC code tends to have slightly higher memory overhead.

I think you will get the same errors on x86 if you raise the --vm or --vm-bytes parameters high enough.  Depending on how much higher it needs to go for x86 we might have a problem, or it might be NOTABUG.

Comment 9 Min Deng 2017-12-14 09:24:57 UTC
(In reply to David Gibson from comment #8)
> I'm assuming those call traces are from the guest not the host?
  The call trace is from guest.
> Did the x86 VM have the same available RAM as on POWER?
  To be honest,after communicated with x86 guys it was very hard to obtain similar size of x86 host in a short time.

Comment 10 Min Deng 2017-12-14 09:27:45 UTC
(In reply to Min Deng from comment #9)
> (In reply to David Gibson from comment #8)
> > I'm assuming those call traces are from the guest not the host?
>   The call trace is from guest.
> > Did the x86 VM have the same available RAM as on POWER?
>   To be honest,after communicated with x86 guys it was very hard to obtain
> similar size of x86 host in a short time.
   Anyway,QE will give the results as soon as grab such a host,any issues please let me know,thanks a lot.

Comment 11 David Gibson 2017-12-19 02:36:47 UTC
Ok, I've had a closer look at this.  The stress command with the parameters given should only use ~40G of RAM, so should fit easily in the 80G of the guest, I'm not sure why we're getting OOMs.

In discussion with mdeng, there's some confusion as to the exact hugepage configuration on the guest.  Specifically, the reproduce instructions suggest hugepages were configured on the host and used to back the guest, but were not configured inside the guest itself.  However the error trace in the attachement suggests that hugepages are configured inside the guest.

Min, can you please re-run the test on POWER (for now). Please include:
  - The exact qemu command line
  - The output from cat /proc/meminfo inside the guest immediately before running stress
  - The output from free -h inside the guest immediately before running stress.

Comment 12 Min Deng 2017-12-19 07:54:12 UTC
Builds,
kernel-4.14.0-18.el7a.ppc64le
qemu-kvm-rhev-2.10.0-12.el7.ppc64le

Settings before guest boot up
#mount -t hugetlbfs none /mnt/kvm_hugepage
#echo 40960 > /proc/sys/vm/nr_hugepages
#cat /proc/meminfo | grep -i hugepage

Boot up guest with the following cli,
/usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pseries -nodefaults -vga std -chardev socket,id=serial_id_serial0,path=/tmp/S,server,nowait -device spapr-vty,reg=0x30000000,chardev=serial_id_serial0 -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=rhel75-alt.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,vhost=on -device virtio-net-pci,netdev=net0,id=nic0,mac=52:54:00:13:17:51,bus=pci.0,addr=0x1e -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :11 -rtc base=utc,clock=host -enable-kvm -device usb-kbd,id=input0 -device usb-mouse,id=input1 -device usb-tablet,id=input2 -monitor stdio -smp 4 -m 80G,slots=256,maxmem=1T -mem-path /mnt/kvm_hugepage -qmp tcp:0:4444,server,nowait -monitor unix:/tmp/monitor3,server,nowait

Case 1,
1. cat /proc/meminfo
[root@localhost home]# cat /proc/meminfo
MemTotal:       81673216 kB
MemFree:        80737792 kB
MemAvailable:   80628544 kB
2.stress --vm 160 --vm-bytes 256M --timeout 30s
stress: info: [9734] dispatching hogs: 0 cpu, 0 io, 160 vm, 0 hdd
stress: info: [9734] successful run completed in 30s
3.[root@localhost home]# free -h
              total        used        free      shared  buff/cache   available
Mem:            77G        359M         76G         21M        556M         76G
Swap:          3.0G          0B        3.0G

Case 2,
1.allocate hugepages within guest.
[root@localhost ~]# mount -t hugetlbfs none /mnt/kvm_hugepage
[root@localhost ~]# echo 40960 > /proc/sys/vm/nr_hugepages
[root@localhost ~]# cat /proc/meminfo | grep -i hugepage
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:   39544
HugePages_Free:    39544
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

cat /proc/meminfo
MemTotal:       81673216 kB
MemFree:          377728 kB
MemAvailable:     526784 kB
Buffers:            4928 kB
Cached:           746176 kB
SwapCached:            0 kB
Active:           493440 kB
Inactive:         502656 kB
Active(anon):     133056 kB
Inactive(anon):   133504 kB
Active(file):     360384 kB
Inactive(file):   369152 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       3145664 kB
SwapFree:        3145664 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        245248 kB
Mapped:            60416 kB
Shmem:             21568 kB
Slab:             139008 kB
SReclaimable:      49472 kB
SUnreclaim:        89536 kB
KernelStack:        2592 kB
PageTables:         4544 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3973568 kB
Committed_AS:     658176 kB
VmallocTotal:   549755813888 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:   39071
HugePages_Free:    39071
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB


[root@localhost home]# free -h
free -h
              total        used        free      shared  buff/cache   available
Mem:            77G         76G        365M         21M        870M        512M
Swap:          3.0G          0B        3.0G

2.run stress test within guest.
[root@localhost home]# stress --vm 160 --vm-bytes 256M --timeout 30s
stress: info: [9613] dispatching hogs: 0 cpu, 0 io, 160 vm, 0 hdd
stress: info: [9613] successful run completed in 30s

result 1,
[root@localhost ~]# stress --vm 160 --vm-bytes 256M --timeout 30s
stress: info: [9849] dispatching hogs: 0 cpu, 0 io, 160 vm, 0 hdd
stress: FAIL: [9849] (416) <-- worker 10009 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9853 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9857 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9859 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9871 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9872 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9879 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9880 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9882 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9885 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9886 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9889 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9890 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9891 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9892 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9897 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9902 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9904 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes
stress: FAIL: [9849] (416) <-- worker 9905 got signal 9
stress: WARN: [9849] (418) now reaping child worker processes

result 2,
From dmesg,
[  207.552033] stress: page allocation stalls for 27450ms, order:0, mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null)
[  207.552711] stress cpuset=/ mems_allowed=0
[  207.553380] CPU: 3 PID: 10608 Comm: stress Not tainted 4.14.0-18.el7a.ppc64le #1
[  207.554054] Call Trace:
[  207.554730] [c0000013cad67870] [c000000000c391cc] dump_stack+0xb0/0xf4 (unreliable)
[  207.555430] [c0000013cad678b0] [c0000000003270cc] warn_alloc+0x12c/0x1d0
[  207.556108] [c0000013cad67950] [c000000000327a3c] __alloc_pages_nodemask+0x7fc/0x1070
[  207.556776] [c0000013cad67b50] [c0000000003d62b0] alloc_pages_vma+0xe0/0x570
[  207.557435] [c0000013cad67bc0] [c000000000381a4c] do_anonymous_page+0x15c/0x990
[  207.558097] [c0000013cad67c30] [c000000000389ebc] __handle_mm_fault+0xc5c/0x1010
[  207.558747] [c0000013cad67d30] [c00000000038a398] handle_mm_fault+0x128/0x200
[  207.559397] [c0000013cad67d70] [c000000000077ac4] do_page_fault+0x1d4/0x6f0
[  207.560061] [c0000013cad67e30] [c00000000000a4d4] handle_page_fault+0x18/0x38
[  208.092055] stress: page allocation stalls for 27840ms, order:0, mode:0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=(null)
[  208.093415] stress cpuset=/ mems_allowed=0
[  208.094090] CPU: 2 PID: 10556 Comm: stress Not tainted 4.14.0-18.el7a.ppc64le #1
[  208.094775] Call Trace:
[  208.095454] [c0000013ed2db620] [c000000000c391cc] dump_stack+0xb0/0xf4 (unreliable)
[  208.096156] [c0000013ed2db660] [c0000000003270cc] warn_alloc+0x12c/0x1d0
[  208.096860] [c0000013ed2db700] [c000000000327a3c] __alloc_pages_nodemask+0x7fc/0x1070
[  208.097548] [c0000013ed2db900] [c0000000003d42e4] alloc_pages_current+0x94/0x250
[  208.098233] [c0000013ed2db950] [c0000000003e5760] new_slab+0x720/0x730
[  208.098921] [c0000013ed2dba10] [c0000000003e8844] ___slab_alloc+0x4f4/0x650
[  208.099610] [c0000013ed2dbb30] [c0000000003ef4a8] __slab_alloc+0x5c/0x88
[  208.100307] [c0000013ed2dbb90] [c0000000003e8ac8] kmem_cache_alloc+0x128/0x2f0
[  208.100997] [c0000013ed2dbbe0] [c000000000386a10] __pmd_alloc+0x80/0x170
[  208.101660] [c0000013ed2dbc30] [c00000000038a1ec] __handle_mm_fault+0xf8c/0x1010
[  208.102312] [c0000013ed2dbd30] [c00000000038a398] handle_mm_fault+0x128/0x200
[  208.102945] [c0000013ed2dbd70] [c000000000077ac4] do_page_fault+0x1d4/0x6f0
[  208.103564] [c0000013ed2dbe30] [c00000000000a4d4] handle_page_fault+0x18/0x38

3.check free -h
[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:            77G         77G        327M        5.1M        151M         28M
Swap:          3.0G        267M        2.7G


  QE got two different results so far,for result 1 was more friendly since we could know the reason why it failed to some extents.For result 2,user thought it passed the test but it didn't except for checking dmesg log.It's a little tricky.Thanks a lot.Any issues please let me know.

Min

Comment 13 David Gibson 2017-12-20 03:34:44 UTC
The failures in case 2 from comment 12 come because nearly all of the guest memory is being occupied by hugepages.  stress needs normal pages to operate, so it causes out of memory errors, the exact details depending on non-deterministic details.

The guess is that the failure in comment 0 occurred for the same reason: hugepages accidentally allocated in the guest as well as the host.

So, I don't think there's actually a bug here.


Note You need to log in before you can comment on or make changes to this bug.