Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1513990 - [virtio-win][whql][balloon] Job DP WLK - Hot-Add - Device Test - Verify driver support for Hot-Add CPU failed on Win2008-r2
Summary: [virtio-win][whql][balloon] Job DP WLK - Hot-Add - Device Test - Verify drive...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: virtio-win
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Gal Hammer
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1451959
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-16 12:48 UTC by Peixiu Hou
Modified: 2019-02-19 10:46 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)
job Verify driver support for Hot-Add CPU failed hck log (deleted)
2017-11-16 15:23 UTC, Peixiu Hou
no flags Details
143 balloon CPU HotAdd case failed latest hck log (deleted)
2017-11-30 09:42 UTC, Peixiu Hou
no flags Details
DP WLK Hot Add test with virtio-win-prewhql-162 --q35 (deleted)
2018-12-07 05:56 UTC, Peixiu Hou
no flags Details
DP WLK Hot Add test with virtio-win-prewhql-162 --pc (deleted)
2018-12-07 05:57 UTC, Peixiu Hou
no flags Details

Description Peixiu Hou 2017-11-16 12:48:37 UTC
Description of problem:

The job "DP WLK - Hot-Add - Device Test - Verify driver support for Hot-Add CPU" failed with follow error on Win2008-r2.

WDTF_SIMPLE_IO : - PerformIO(QEMU QEMU DVD-ROM ATA Device (E:\) IDE\CDROMQEMU_QEMU_DVD-ROM_______________________2.5+____\5&3A2A5854&0&1.0.0) Failed : Opening 'E:\sources\en-us\setup.exe.mui' file. Win32=999 - Error performing inpage operation. Count 88826


Version-Release number of selected component (if applicable):
kernel-3.10.0-774.el7.x86_64
qemu-kvm-rhev-2.10.0-6.el7.x86_64.rpm
virtio-win-prewhql-143
seabios-bin-1.10.2-5.el7

How reproducible:
100%

Steps to Reproduce:
1. Boot guest with command:
/usr/libexec/qemu-kvm -name 143BLN2008R207Z -enable-kvm -m 4G -smp 4 -uuid 8f337e59-bc44-43c2-a9e5-b2de474e57c5 -nodefconfig -nodefaults -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -chardev socket,id=charmonitor,path=/tmp/143BLN2008R207Z,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb -drive file=143BLN2008R207Z,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=143BLN2008R207Z.vfd,if=floppy,id=drive-fdc0-0-0,format=raw,cache=none -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:41:17:ee:5d -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:2 -vga std -M pc -device virtio-balloon-pci,id=balloon0,bus=pci.0

2. Run the job "DP WLK - Hot-Add - Device Test - Verify driver support for Hot-Add CPU" in HCK stdio.

3. Check the job result

Actual results:
Failed with error

Expected results:
Pass

Additional info:

1. Cannot reproduce this bug with qemu-kvm-rhev-2.9.0-2.el7.x86_64 + seabios-bin-1.10.2-2.el7.noarch  + virtio-win-prewhql-137, without hv- related flags.
2. Reproduced this bug with qemu-kvm-rhev-2.9.0-2.el7.x86_64 + seabios-bin-1.10.2-2.el7.noarch + virtio-win-prewhql-143, without hv- related flags.
3. Cannot reproduce this bug without en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso cdrom.
4. Reproduced this bug with qemu-kvm-rhev-2.9.0-5.el7.x86_64 + seabios-bin-1.10.2-5.el7.noarch + virtio-win-prewhql-143.

Comment 2 Peixiu Hou 2017-11-16 15:23:27 UTC
Created attachment 1353575 [details]
job Verify driver support for Hot-Add CPU failed hck log

Comment 3 Peixiu Hou 2017-11-17 07:18:03 UTC
Tried to rerun this job:

It can be passed with qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64 + virtio-win-prewhql-143 + seabios-bin-1.10.2-5.el7 + hv related flags.

Thanks~
Peixiu

Comment 4 lijin 2017-11-17 07:21:20 UTC
Vadim,

Could you have a check about comment#3?

Is it a dup of bz1513833?

Comment 5 Vadim Rozenfeld 2017-11-17 07:32:32 UTC
(In reply to lijin from comment #4)
> Vadim,
> 
> Could you have a check about comment#3?
> 
> Is it a dup of bz1513833?

They can be quite closely related. 
I would suggest re-check all HCK hv_* related problems on top of 
qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64

Best regards,
Vadim.

Comment 6 lijin 2017-11-17 07:40:56 UTC
(In reply to Vadim Rozenfeld from comment #5)
> (In reply to lijin from comment #4)
> > Vadim,
> > 
> > Could you have a check about comment#3?
> > 
> > Is it a dup of bz1513833?
> 
> They can be quite closely related. 
> I would suggest re-check all HCK hv_* related problems on top of 
> qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64

Sure.
For now, only these two bugs found,we will check with it if more jobs fail.

BTW,will the formal build for qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64 be born out soon? We'd like to run other whql test on this stable build :-)

Comment 7 Peixiu Hou 2017-11-17 07:42:10 UTC
And tried to rerun this job with qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64 + virtio-win-prewhql-143 + seabios-bin-1.10.2-5.el7 + none hv related flags.

It will failed with error message as(tried 1 time):

Error 11/17/2017 2:22:18.390 AM WDTF_SIMPLE_IO : - PerformIO(QEMU QEMU DVD-ROM ATA Device (E:\) IDE\CDROMQEMU_QEMU_DVD-ROM_______________________2.5+____\5&3A2A5854&0&1.0.0) Failed : Opening 'E:\sources\replacementmanifests\capi2_certs-repl.man' file. Win32=999 - Error performing inpage operation. Count 142598 
File:    Line: -1 
Error Type:    
Error Code:   0x0 
Error Text:   Error 0x00000000

Comment 8 Vadim Rozenfeld 2017-11-17 07:54:16 UTC
(In reply to lijin from comment #6)
> (In reply to Vadim Rozenfeld from comment #5)
> > (In reply to lijin from comment #4)
> > > Vadim,
> > > 
> > > Could you have a check about comment#3?
> > > 
> > > Is it a dup of bz1513833?
> > 
> > They can be quite closely related. 
> > I would suggest re-check all HCK hv_* related problems on top of 
> > qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64
> 
> Sure.
> For now, only these two bugs found,we will check with it if more jobs fail.
> 
> BTW,will the formal build for qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64 be
> born out soon? We'd like to run other whql test on this stable build :-)

Yes, going to send the patch today.
Vadim.

Comment 9 Peixiu Hou 2017-11-17 14:05:04 UTC
Tired to retest this case without hv- related flags, the result as follows:

1. With qemu-kvm-rhev-2.9.0-2.el7.x86_64 + virtio-win-prewhql-137 --Passed(2/2)
2. With qemu-kvm-rhev-2.9.0-2.el7.x86_64 + virtio-win-prewhql-143 --Failed(3/4)
The error log as:
-------------------------------------------------------------------------------
Error 11/17/2017 6:27:27.531 AM WDTF_SIMPLE_IO : - PerformIO(QEMU QEMU DVD-ROM ATA Device (E:\) IDE\CDROMQEMU_QEMU_DVD-ROM_______________________2.5+____\5&3A2A5854&0&1.0.0) Failed : Reading 'E:\sources\servicingstackmisc\apds.dll' file. Win32=999 - Error performing inpage operation. Count 125591 
File:    Line: -1 
Error Type:    
Error Code:   0x0 
Error Text:   Error 0x00000000 
-------------------------------------------------------------------------------

The guest cmd as:
/usr/libexec/qemu-kvm -name 143BLN2008R207Z -enable-kvm -m 4G -smp 4 -uuid 8f337e59-bc44-43c2-a9e5-b2de474e57c5 -nodefconfig -nodefaults -cpu host -chardev socket,id=charmonitor,path=/tmp/143BLN2008R207Z,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb -drive file=143BLN2008R207Z,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=143BLN2008R207Z.vfd,if=floppy,id=drive-fdc0-0-0,format=raw,cache=none -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:41:17:ee:5d -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:5 -vga std -M pc -device virtio-balloon-pci,id=balloon0,bus=pci.0

Thanks~
Peixiu Hou

Comment 10 Peixiu Hou 2017-11-29 06:45:59 UTC
Tested this case with the latest version qemu-kvm-common-rhev-2.10.0-8.el7.x86_64:

1. Tested with hv- flags, run 5 times, reproduced this bug 3 times, passed 2 times.
2. Tested without hv- flags, run 2 times. reproduced this bug 1 time, passed 1 time.

Used other versions:
kernel-3.10.0-774.el7.x86_64
virtio-win-prewhql-143
seabios-1.10.2-5.el7.x86_64

Best Regards~
Peixiu Hou

Comment 11 lijin 2017-11-29 06:51:31 UTC
Hi Gal and Vadim,

Could you have a check of comment#10, seems the new qemu version does not fix this issue.

Comment 12 lijin 2017-11-29 06:51:57 UTC
Hi Gal and Vadim,

Could you have a check of comment#10, seems the new qemu version does not fix this issue.

Comment 13 Vadim Rozenfeld 2017-11-29 08:38:56 UTC
That's quite interesting, because it looks like WDTF fails to run simple io test against CD-ROM. what happens if you remove cdrom sting from qemu command line. 
Could you also upload the latest hck log file?

Thanks,
Vadim.

Comment 14 Peixiu Hou 2017-11-30 09:40:22 UTC
(In reply to Vadim Rozenfeld from comment #13)
> That's quite interesting, because it looks like WDTF fails to run simple io
> test against CD-ROM. what happens if you remove cdrom sting from qemu
> command line. 
> Could you also upload the latest hck log file?
> 
Hi Vadim,

Tried to test after removed cdrom string from qemu command line, it can be passed as comment#0 "Additional info -3" mentioned .

The latest hck log file pls refer to attachment.

Thanks~
Peixiu


> Thanks,
> Vadim.

Comment 15 Peixiu Hou 2017-11-30 09:42:07 UTC
Created attachment 1360775 [details]
143 balloon CPU HotAdd case failed latest hck log

Comment 16 lijin 2018-06-27 03:37:37 UTC
tried 10 times with build 154(cdrom attached), only 5 of them passed.
tried 10 times with build 154 without cdrom, 100% pass

qemu cli:
/usr/libexec/qemu-kvm -name 154BLNW10S64SXG -enable-kvm -m 6G -smp 8 -uuid 78219151-e8dc-46c6-9e30-3e6792643350 -nodefconfig -nodefaults -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -chardev socket,id=charmonitor,path=/tmp/154BLNW10S64SXG,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb -drive file=154BLN2008R2MWN,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,script=/etc/qemu-ifup1,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:6e:3e:cc:e8 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga std -M pc -device virtio-balloon-pci,id=balloon0,bus=pci.0 -monitor stdio -drive file=/home/kvm_autotest_root/iso/ISO/Win2008R2/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0

Comment 18 Peixiu Hou 2018-12-07 05:54:41 UTC
Also hit this job failed with virtio-win-prewhql-162, but not 100% reproduced.

1. Tested this job 5 times with machine type q35 on rhel8.0 host, hit 1 time failed, 4 times passed.
2. Tested this job 5 times with machine type pc on rhel8.0 host, hit 4 times fialed, 1 time passed.

The failed log is not same as comment#0, as follows:
================================================================================
 Failed 
 Start Test 12/6/2018 6:16:18.593 AM WDTF_TEST : Perform 1 test cycles of Device I/O 
Message 12/6/2018 6:16:18.593 AM WDTF_TEST : Device I/O test Cycle #1 
Message 12/6/2018 6:16:18.593 AM WDTF_SIMPLEIO_STRESS : - Start(QEMU QEMU DVD-ROM ATA Device (E:\) IDE\CDROMQEMU_QEMU_DVD-ROM_______________________2.5+____\5&3A2A5854&0&1.0.0) 
Message 12/6/2018 6:16:18.593 AM WDTF_SIMPLEIO_STRESS : - Start(Generic volume (D:\) STORAGE\VOLUME\{937DAF86-F9DE-11E8-8F3B-806E6F6E6963}#0000000000100000) 
Message 12/6/2018 6:16:18.593 AM WDTF_SIMPLEIO_STRESS : - Start(Generic volume (C:\) STORAGE\VOLUME\{937DAF86-F9DE-11E8-8F3B-806E6F6E6963}#0000000006500000) 
Message 12/6/2018 6:16:19.593 AM WDTF_SUPPORT : - WaitForMinutes : 20 
Message 12/6/2018 6:36:19.593 AM WDTF_SIMPLEIO_STRESS : - Stop(QEMU QEMU DVD-ROM ATA Device (E:\) IDE\CDROMQEMU_QEMU_DVD-ROM_______________________2.5+____\5&3A2A5854&0&1.0.0) 
Message 12/6/2018 6:36:20.593 AM WDTF_SIMPLEIO_STRESS : - Stop(Generic volume (D:\) STORAGE\VOLUME\{937DAF86-F9DE-11E8-8F3B-806E6F6E6963}#0000000000100000) 
Message 12/6/2018 6:36:20.593 AM WDTF_SIMPLEIO_STRESS : - Stop(Generic volume (C:\) STORAGE\VOLUME\{937DAF86-F9DE-11E8-8F3B-806E6F6E6963}#0000000006500000) 
End Test 12/6/2018 6:36:20.593 AM WDTF_TEST : Perform 1 test cycles of Device I/O 
Result:   Fail 
================================================================================ 

The latest HCK log please refer to attachment.

Comment 19 Peixiu Hou 2018-12-07 05:56:51 UTC
Created attachment 1512382 [details]
DP WLK Hot Add test with virtio-win-prewhql-162  --q35

Comment 20 Peixiu Hou 2018-12-07 05:57:44 UTC
Created attachment 1512383 [details]
DP WLK Hot Add test with virtio-win-prewhql-162  --pc


Note You need to log in before you can comment on or make changes to this bug.