Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1515663 - [virtio-win][whql][NETKVM] guest bsod(8e) when running some jobs on win7-32 --DF-PNP & DF-Fuzz &CHAOS
Summary: [virtio-win][whql][NETKVM] guest bsod(8e) when running some jobs on win7-32 -...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Yan Vugenfirer
QA Contact: Peixiu Hou
URL:
Whiteboard:
: 1577716 (view as bug list)
Depends On:
Blocks: 1644499
TreeView+ depends on / blocked
 
Reported: 2017-11-21 08:11 UTC by Peixiu Hou
Modified: 2019-03-03 09:50 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1644499 (view as bug list)
Environment:
Last Closed: 2019-02-28 19:34:33 UTC


Attachments (Terms of Use)
dump windbg info of each hit BSOD(8e) job (deleted)
2018-05-11 06:55 UTC, Peixiu Hou
no flags Details
content for Arg3: 8453dc80 (deleted)
2018-05-23 09:44 UTC, Peixiu Hou
no flags Details
content for Arg4: 8453d900 (deleted)
2018-05-23 09:44 UTC, Peixiu Hou
no flags Details
content for Arg2: 84d072b0 (deleted)
2018-05-23 09:45 UTC, Peixiu Hou
no flags Details
WinDbg_printout_messages (deleted)
2018-05-23 09:46 UTC, Peixiu Hou
no flags Details

Description Peixiu Hou 2017-11-21 08:11:36 UTC
Description of problem:

Hit the BSOD(8e) when running follow jobs:

DF - Concurrent Hardware And Operating System (CHAOS) Test (Certification)
DF - PNP Remove Device Test (Certification)
DF - PNP Cancel Stop Device Test (Certification)
DF - Reinstall with IO Before and After (Certification)
DF - Fuzz Query and Set File Information Test (Certification)
DF - Fuzz random FSCTL test (Certification)
DF - Fuzz open and close test (Certification)


Version-Release number of selected component (if applicable):
kernel-3.10.0-774.el7.x86_64
qemu-kvm-rhev-2.10.0-6.el7.x86_64.rpm
virtio-win-prewhql-143
seabios-bin-1.10.2-5.el7

How reproducible:
100%

Steps to Reproduce:
1.Boot the guest up:
/usr/libexec/qemu-kvm -name 143NICWIN732CZP -enable-kvm -m 2G -smp 2 -uuid 111bb838-ef99-457b-ae10-6cfa26a49783 -nodefconfig -nodefaults -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -chardev socket,id=charmonitor,path=/tmp/143NICWIN732CZP,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb -drive file=143NICWIN732CZP,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=en_windows_7_ultimate_with_sp1_x86_dvd_u_677460.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=143NICWIN732CZP.vfd,if=floppy,id=drive-fdc0-0-0,format=raw,cache=none -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:1b:29:0f:e0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -vga std -M pc -netdev tap,script=/etc/qemu-ifup-private,downscript=no,id=hostnet1,vhost=on,queues=2 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=00:52:03:57:6f:b8,bus=pci.0,mq=on,vectors=6
2. Submit the job from HCK stdio.
3. Check the guest status

Actual results:
BSOD(8e)

Expected results:
Passed without BSOD

Additional info:
1. Tried to test with the qemu-kvm-rhev-2.10.0-5.el7.bz1451959.x86_64, this bug can be reproduced.

Comment 2 Peixiu Hou 2017-11-21 08:12:50 UTC
Windebug info:

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000420, 8295720c, 8234b974, 0}

Page 1a5af not present in the dump file. Type ".hh dbgerr004" for details
Probably caused by : ntkrpamp.exe ( nt!ViCtxCheckAndReleaseXSaveData+59 )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000420, The exception code that was not handled
Arg2: 8295720c, The address that the exception occurred at
Arg3: 8234b974, Trap Frame
Arg4: 00000000

Debugging Details:
------------------

Page 1a5af not present in the dump file. Type ".hh dbgerr004" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000420 - An assertion failure has occurred.

FAULTING_IP: 
nt!ViCtxCheckAndReleaseXSaveData+59
8295720c cd2c            int     2Ch

TRAP_FRAME:  8234bab4 -- (.trap 0xffffffff8234bab4)
ErrCode = 00000010
eax=89672f01 ebx=823f7744 ecx=102a1001 edx=00f800c0 esi=02000000 edi=00200000
eip=84ab700a esp=8234bb28 ebp=8234bbdc iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
Ntfs!NtfsCommonCreate+0x2f9:
84ab700a f6c101          test    cl,1
Resetting default scope

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x8E

PROCESS_NAME:  Te.exe

CURRENT_IRQL:  9

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

EXCEPTION_RECORD:  8aba7a00 -- (.exr 0xffffffff8aba7a00)
ExceptionAddress: 8295707c (nt!ViCtxIsr)
   ExceptionCode: 02780016
  ExceptionFlags: 8aba7a04
NumberParameters: 0

LAST_CONTROL_TRANSFER:  from 826cb08c to 826f4f20

STACK_TEXT:  
8234b4e4 826cb08c 0000008e c0000420 8295720c nt!KeBugCheckEx+0x1e
8234b904 82654dd6 8234b920 00000000 8234b974 nt!KiDispatchException+0x1ac
8234b96c 82654d72 8234b9f4 8295720e badb0d00 nt!CommonDispatchException+0x4a
8234b994 826db07f 8270df10 00000000 00000000 nt!KiExceptionExit+0x17a
8234b9f4 829570ac 84ca12b0 8aba7a00 8ab7f610 nt!vDbgPrintExWithPrefixInternal+0x2ff
8234ba14 826507ad 8aba7a00 8ab8a428 8234ba40 nt!ViCtxIsr+0x30
8234ba14 826573b8 8aba7a00 8ab8a428 8234ba40 nt!KiInterruptDispatch+0x6d
8234bab4 84ab700a badb0d00 00f800c0 8234bb00 nt!KiTrap0E+0xbc
8234bbdc 84a3b09c 98953a90 89672e28 823f7744 Ntfs!NtfsCommonCreate+0x2f9
8234bc1c 826957ba 823f76dc 00000000 ffffffff Ntfs!NtfsCommonCreateCallout+0x20
8234bc1c 826958b1 823f76dc 00000000 ffffffff nt!KiSwapKernelStackAndExit+0x15a
823f7640 8269f7c7 823f76dc 84a3b07c 8234c000 nt!KiSwitchKernelStackAndCallout+0x31
823f76b4 84a3afd2 84a3b07c 823f76dc 00000000 nt!KeExpandKernelStackAndCalloutEx+0x29d
823f76ec 84acca2a 98953a90 89672e28 823f7744 Ntfs!NtfsCommonCreateOnNewStack+0x39
823f77e8 829476c3 8a4c0020 89672e28 00000000 Ntfs!NtfsFsdCreate+0x1f8
823f780c 8264d54a 00000000 89672e28 8a4c0020 nt!IovCallDriver+0x258
823f7820 847d020c 89672e28 00000000 89672fdc nt!IofCallDriver+0x1b
823f7844 847e38c9 823f7864 8a4bf7d8 00000000 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2aa
823f7890 829476c3 8a4bf7d8 8a4bf298 89a10788 fltmgr!FltpCreate+0x2db
823f78b4 8264d54a 00000000 89a107e4 8a4bf7d8 nt!IovCallDriver+0x258
823f78c8 8285d2a9 d7741585 823f7a70 00000000 nt!IofCallDriver+0x1b
823f79a0 8283cac5 8a44ba08 c55c0040 894a7d20 nt!IopParseDevice+0xed7
823f7a1c 8284ced6 00000000 823f7a70 00000040 nt!ObpLookupObjectName+0x4fa
823f7a78 8286fac3 00a4ecc0 845c0040 00000401 nt!ObOpenObjectByName+0x165
823f7c24 826541ea 00a4ecc0 00a4ec98 00a4ecf0 nt!NtQueryAttributesFile+0x121
823f7c24 775470b4 00a4ecc0 00a4ec98 00a4ecf0 nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
00a4ecf0 00000000 00000000 00000000 00000000 0x775470b4


STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!ViCtxCheckAndReleaseXSaveData+59
8295720c cd2c            int     2Ch

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  nt!ViCtxCheckAndReleaseXSaveData+59

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrpamp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce78a09

IMAGE_VERSION:  6.1.7601.17514

FAILURE_BUCKET_ID:  0x8E_VRF_nt!ViCtxCheckAndReleaseXSaveData+59

BUCKET_ID:  0x8E_VRF_nt!ViCtxCheckAndReleaseXSaveData+59

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x8e_vrf_nt!victxcheckandreleasexsavedata+59

FAILURE_ID_HASH:  {c54dfd20-598e-a97b-7063-6b1ad9173a38}

Followup: MachineOwner
---------

Comment 12 Peixiu Hou 2017-12-22 10:15:01 UTC
For this issue, I also done follow tests:

1. Tried test this case with a manually installed image, also can reproduced this bug, tested with build 143 version.
2. Tried tested with build 142 version, also can reproduced this issue, tested 3 times, the job is passed at the first and second time, at the third time, hit BSOD(8e). 

Used versions:
kernel-3.10.0-820.el7.x86_64
qemu-kvm-rhev-2.9.0-14.el7.x86_64
seabios-1.11.0-1.el7.x86_64

Best Regards~
Peixiu

Comment 13 Peixiu Hou 2017-12-22 10:16:30 UTC
Only tested one job: DF - Fuzz random IOCTL test (Certification)

Comment 17 Peixiu Hou 2017-12-26 09:59:41 UTC
Hi,

Tried to run this job on e1000 device(select the e1000 on the hck stdio, delete the netkvm driver and remove the virtio-net device from the command line), also can reproduce this issue.

On the old image, reproduced 1/8.
On the new installed image, reproduced 5/5.

Used versions:
kernel-3.10.0-820.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7.x86_64
seabios-1.11.0-1.el7.x86_64

Also will try to run on rhel7.4 version, will update the result here once get the result.

Thanks~
Peixiu

Comment 18 Sameeh Jubran 2017-12-26 13:03:09 UTC
(In reply to Peixiu Hou from comment #17)
> Hi,
> 
> Tried to run this job on e1000 device(select the e1000 on the hck stdio,
> delete the netkvm driver and remove the virtio-net device from the command
> line), also can reproduce this issue.
Great, so we can say that this is not related to NetKVM driver and it is some other issue.

I have tried to reproduce with the Windows 7 image you provided me with but couldn't. However Windows had run chkdsk durig boot and fixed many errors with the file system! Can you try and run chkdsk and see if this reproduces?

> 
> On the old image, reproduced 1/8.
> On the new installed image, reproduced 5/5.
> 
> Used versions:
> kernel-3.10.0-820.el7.x86_64
> qemu-kvm-rhev-2.10.0-12.el7.x86_64
> seabios-1.11.0-1.el7.x86_64
> 
> Also will try to run on rhel7.4 version, will update the result here once
> get the result.
> 
> Thanks~
> Peixiu

Comment 19 Peixiu Hou 2017-12-28 09:52:44 UTC
(In reply to Sameeh Jubran from comment #18)
> (In reply to Peixiu Hou from comment #17)
> > Hi,
> > 
> > Tried to run this job on e1000 device(select the e1000 on the hck stdio,
> > delete the netkvm driver and remove the virtio-net device from the command
> > line), also can reproduce this issue.
> Great, so we can say that this is not related to NetKVM driver and it is
> some other issue.
> 
> I have tried to reproduce with the Windows 7 image you provided me with but
> couldn't. However Windows had run chkdsk durig boot and fixed many errors
> with the file system! Can you try and run chkdsk and see if this reproduces?
> 
Hi, I tried to run chkdsk on the guest after hit this issue, the chkdsk run finished successfully, shown "no problem found", and on this progress, not hit the BSOD.
After the chkdsk finished, rerun this job, also can reproduced this bug(with e1000 device).

Thanks~
Peixiu

Comment 20 Peixiu Hou 2018-01-03 09:28:45 UTC
Tried test this issue on rhel7.4 host, it also can be reproduced(2/3).

Used versions:
kernel-3.10.0-693.el7.x86_64.rpm
qemu-kvm-rhev-2.9.0-14.el7.x86_64.rpm
seabios-1.10.2-4.el7_4.x86_64.rpm
virtio-win-prewhql-141

Comment 21 lijin 2018-01-09 07:24:07 UTC
Should we move the component to qemu if it's not related to virtio-win drivers?

Comment 22 Yan Vugenfirer 2018-01-09 09:45:11 UTC
I think so.

Comment 23 lijin 2018-01-09 10:19:45 UTC
change component to qemu-kvm-rhev according to commet#17,18,22

Comment 29 Peixiu Hou 2018-05-11 06:55:17 UTC
Created attachment 1434758 [details]
dump windbg info of each hit BSOD(8e) job

Hi Yan,

sorry for misunderstood your meaning, other cases failed with the same BSOD id(8e), but the stack in the crash dumps looks not same. 

For follow 3 jobs, dump with same "MODULE_NAME: nt IMAGE_NAME:  ntkrpamp.exe":
1.DF - Concurrent Hardware And Operating System (CHAOS) Test (Certification)
2.DF - PNP Remove Device Test (Certification)
3.DF - PNP Cancel Stop Device Test (Certification)

For follow 2 jobs, dump with same "MODULE_NAME: win32k IMAGE_NAME:  win32k.sys"
6.DF - Fuzz random FSCTL test (Certification)
7.DF - Fuzz open and close test (Certification)

For the job "DF - Fuzz Query and Set File Information Test (Certification)", dump with "MODULE_NAME: intelppm IMAGE_NAME:  intelppm.sys".

For more details information please refer to the pasted text.

For the job "DF - Reinstall with IO Before and After (Certification)" and "DF - Fuzz open and close test (Certification)", cannot reproduce now, the dump saved before with "MODULE_NAME: Unknown_Module IMAGE_NAME:  Unknown_Image", should be invalid, but also pasted. 

Best Regards~
Peixiu

Comment 30 Yan Vugenfirer 2018-05-13 11:07:38 UTC
(In reply to Peixiu Hou from comment #29)
> Created attachment 1434758 [details]
> dump windbg info of each hit BSOD(8e) job
> 
> Hi Yan,
> 
> sorry for misunderstood your meaning, other cases failed with the same BSOD
> id(8e), but the stack in the crash dumps looks not same. 
> 
> For follow 3 jobs, dump with same "MODULE_NAME: nt IMAGE_NAME: 
> ntkrpamp.exe":
> 1.DF - Concurrent Hardware And Operating System (CHAOS) Test (Certification)
> 2.DF - PNP Remove Device Test (Certification)
> 3.DF - PNP Cancel Stop Device Test (Certification)
> 
> For follow 2 jobs, dump with same "MODULE_NAME: win32k IMAGE_NAME: 
> win32k.sys"
> 6.DF - Fuzz random FSCTL test (Certification)
> 7.DF - Fuzz open and close test (Certification)
> 
> For the job "DF - Fuzz Query and Set File Information Test (Certification)",
> dump with "MODULE_NAME: intelppm IMAGE_NAME:  intelppm.sys".
> 
> For more details information please refer to the pasted text.
> 

Please upload other memory dumps as well. Thanks!

> For the job "DF - Reinstall with IO Before and After (Certification)" and
> "DF - Fuzz open and close test (Certification)", cannot reproduce now, the
> dump saved before with "MODULE_NAME: Unknown_Module IMAGE_NAME: 
> Unknown_Image", should be invalid, but also pasted. 
> 
> Best Regards~
> Peixiu

Comment 45 Yan Vugenfirer 2018-05-17 09:43:57 UTC
Can it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1421583#c9 ?

Radim, can you take a look at the dump above?

Comment 50 Yan Vugenfirer 2018-05-21 06:23:48 UTC
Peixiu - another thing, can you please post "cat /proc/cpuinfo" from the host. I cannot reproduce the bu on my machines, interesting to see what's the difference.

Comment 51 Peixiu Hou 2018-05-21 06:43:42 UTC
(In reply to Yan Vugenfirer from comment #50)
> Peixiu - another thing, can you please post "cat /proc/cpuinfo" from the
> host. I cannot reproduce the bu on my machines, interesting to see what's
> the difference.

Hi Yan, paste the cpuinfo first, will try to test with WinDbg attached, once get the result will paste to the BZ.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
stepping	: 4
microcode	: 0x200003a
cpu MHz		: 2100.000
cache size	: 16896 KB
physical id	: 0
siblings	: 24
core id		: 0
cpu cores	: 12
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku ospke spec_ctrl intel_stibp
bogomips	: 4200.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:
...
...
...
processor	: 47
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
stepping	: 4
microcode	: 0x200003a
cpu MHz		: 2100.000
cache size	: 16896 KB
physical id	: 1
siblings	: 24
core id		: 11
cpu cores	: 12
apicid		: 55
initial apicid	: 55
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku ospke spec_ctrl intel_stibp
bogomips	: 4204.46
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

Comment 52 Yan Vugenfirer 2018-05-21 07:15:13 UTC
Thanks! It looks your server has an additional "xsavec" flag.

Comment 54 Yan Vugenfirer 2018-05-22 11:27:40 UTC
*** Bug 1577716 has been marked as a duplicate of this bug. ***

Comment 60 Peixiu Hou 2018-05-23 09:44:00 UTC
Created attachment 1440523 [details]
content for Arg3: 8453dc80

Comment 61 Peixiu Hou 2018-05-23 09:44:41 UTC
Created attachment 1440524 [details]
content for Arg4: 8453d900

Comment 62 Peixiu Hou 2018-05-23 09:45:38 UTC
Created attachment 1440531 [details]
content for Arg2: 84d072b0

Comment 63 Peixiu Hou 2018-05-23 09:46:43 UTC
Created attachment 1440538 [details]
WinDbg_printout_messages

Comment 66 Yan Vugenfirer 2018-05-28 09:33:27 UTC
Hi Peixiu,

Can you please run the following during crash (the second parameter is the size)

d <Address pointed by ARG3 of the crash> LA7E

d <Address pointed by ARG4 of the crash> LA7E

Thanks.

Comment 69 Eduardo Habkost 2018-05-28 18:12:51 UTC
Byte 512 is the first thing the nt!RtlEqualExtendedState function checks, so that's really the root cause of the problem:

nt!RtlEqualExtendedState:
82740945 8bff            mov     edi,edi
82740947 55              push    ebp
82740948 8bec            mov     ebp,esp
8274094a 83ec10          sub     esp,10h
8274094d 53              push    ebx
8274094e 8b5d0c          mov     ebx,dword ptr [ebp+0Ch]   # EBX = ARG0
82740951 56              push    esi
82740952 b8e003dfff      mov     eax,0FFDF03E0h            # EAX = STATIC_VAR
82740957 8b30            mov     esi,dword ptr [eax]       # ESI = STATIC_VAR[0]
82740959 57              push    edi
8274095a 8b7804          mov     edi,dword ptr [eax+4]     # EDI = STATIC_VAR[4]
8274095d 8b4508          mov     eax,dword ptr [ebp+8]     # EAX = ARG1
82740960 8b9004020000    mov     edx,dword ptr [eax+204h]  # EDX = ARG1[XCOMP_BV]
82740966 8b8800020000    mov     ecx,dword ptr [eax+200h]  # ECX = ARG1[XSTATE_BV]
8274096c 23d7            and     edx,edi                   # EDX &= STATIC_VAR[4]
8274096e 8955f4          mov     dword ptr [ebp-0Ch],edx   # [ebp-0ch] = ARG1[XCOMP_BV]
82740971 8b9300020000    mov     edx,dword ptr [ebx+200h]  # EDX = ARG0[XSTATE_BV]
82740977 23ce            and     ecx,esi                   # ECX &= STATIC_VAR[0]
82740979 23d6            and     edx,esi                   # EDX &= STATIC_VAR[0]
8274097b 8bb304020000    mov     esi,dword ptr [ebx+204h]  # ESI = ARG0[XCOMP_BV]
82740981 23f7            and     esi,edi
82740983 894df0          mov     dword ptr [ebp-10h],ecx
82740986 3bd1            cmp     edx,ecx                   # (ARG0[XSTATE_BV] & STATIC_VAR[0]) == (ARGV1[XSTATE_BV] & STATIC_VAR[0]) ?
82740988 7505            jne     nt!RtlEqualExtendedState+0x4a (8274098f) (OUT-RETURN0)
[...]
OUT-RETURN0:
8274098f 32c0            xor     al,al
82740991 e9ed000000      jmp     nt!RtlEqualExtendedState+0x13e (82740a83) (OUT-RETURN)
[...]
OUT-RETURN:
82740a83 5f              pop     edi
82740a84 5e              pop     esi
82740a85 5b              pop     ebx
82740a86 c9              leave
82740a87 c20800          ret     8


Can we get a dump of the first 16 bytes of memory address 0FFDF03E0h just to be sure?

Comment 70 Eduardo Habkost 2018-05-28 18:44:53 UTC
Changing XSTATE_BV seems to be correct according to Intel SDM:

"""
13.7 OPERATION OF XSAVE
[...]
If none of these conditions cause a fault, execution of XSAVE reads the XSTATE_BV field of the XSAVE header (see Section 13.4.2) and writes it back to memory, setting XSTATE_BV[i] (0 ≤ i ≤ 63) as follows:
* If RFBM[i] = 0, XSTATE_BV[i] is not changed.
* If RFBM[i] = 1, XSTATE_BV[i] is set to the value of XINUSE[i]. Section 13.6 defines XINUSE to describe the processor init optimization and specifies the initial configuration of each state component. The nature of that optimization implies the following:
— If state component i is in its initial configuration,
  XINUSE[i] may be either 0 or 1, and XSTATE_BV[i] may be
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  written with either 0 or 1.
[...]
"""

SSE state (XMM0-XMM15) is all zeroes on the XSAVE dump, so XINUSE[2] can be 1 or 0.  However, Windows doesn't like it if XSTATE_BV[2] changes from 0 to 1.

We can try to prevent XSTATE_BV[2] from changing from 0 to 1 if XMM0-XMM15 are all zeroes that from happening for the sake of compatibility, but I'm not sure hardware will allow us to do that.

Comment 71 Eduardo Habkost 2018-05-28 20:29:50 UTC
Probably triggered by the following commit, backported on kernel-3.10.0-581.el7.


commit 00c87e9a70a17b355b81c36adedf05e84f54e10d
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Wed Feb 1 14:19:53 2017 +0100

    KVM: x86: do not save guest-unsupported XSAVE state
    
    Saving unsupported state prevents migration when the new host does not
    support a XSAVE feature of the original host, even if the feature is not
    exposed to the guest.
    
    We've masked host features with guest-visible features before, with
    4344ee981e21 ("KVM: x86: only copy XSAVE state for the supported
    features") and dropped it when implementing XSAVES.  Do it again.
    
    Fixes: df1daba7d1cb ("KVM: x86: support XSAVES usage in the host")
    Cc: stable@vger.kernel.org
    Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d153be8929a6..e52c9088660f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3182,6 +3182,7 @@ static void fill_xsave(u8 *dest, struct kvm_vcpu *vcpu)
        memcpy(dest, xsave, XSAVE_HDR_OFFSET);
 
        /* Set XSTATE_BV */
+       xstate_bv &= vcpu->arch.guest_supported_xcr0 | XFEATURE_MASK_FPSSE;
        *(u64 *)(dest + XSAVE_HDR_OFFSET) = xstate_bv;
 
        /*

Comment 72 Eduardo Habkost 2018-05-28 20:32:57 UTC
(In reply to Eduardo Habkost from comment #71)
> Probably triggered by the following commit, backported on
> kernel-3.10.0-581.el7.

Nevermind, I didn't notice the added line was "&=" and not "=".

Comment 73 Eduardo Habkost 2018-05-28 21:11:04 UTC
Radim, any idea on what could be causing xstate_bv unexpectedly change from 1 to 3?

Comment 74 Yan Vugenfirer 2018-05-30 11:40:20 UTC
Another xsave related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1447267#c41

Comment 76 Paolo Bonzini 2018-07-15 08:50:07 UTC
KVM does not, and cannot, trap XSAVE/XRSTOR.  Any behavior that the guest sees is introduced by the processor directly.  I think this is either a Windows bug, or the Driver Verifier should take into account that XINUSE, and therefore XSTATE_BV, can change.

Comment 81 Paolo Bonzini 2018-07-28 11:44:46 UTC
Sorry, the other way round: RHEL uses it, Windows doesn't (see RtlXRestore).  So the test would be to use noxsaves on the kernel command line.

It's possible that more recent Windows versions do use xsavec or xsaves, and that's why the test passes there.

Comment 83 Yan Vugenfirer 2018-11-15 10:40:07 UTC
Hi Yu,

Can you pass the tests with -xsave flag (comment #53 and comment #55).

Comment 84 Yu Wang 2018-11-19 07:12:57 UTC
(In reply to Yan Vugenfirer from comment #83)
> Hi Yu,
> 
> Can you pass the tests with -xsave flag (comment #53 and comment #55).

Yes, all these cases can be pass with "-xsave" flag

Thanks
Yu Wang

Comment 88 Yu Wang 2019-01-28 07:33:10 UTC
cpu info :

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping	: 7
microcode	: 0x714
cpu MHz		: 2314.575
cache size	: 15360 KB
physical id	: 1
siblings	: 12
core id		: 5
cpu cores	: 6
apicid		: 43
initial apicid	: 43
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1d
bogomips	: 4004.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:


Note You need to log in before you can comment on or make changes to this bug.