Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1690637 - Incorrect IO and emulator threads auto-pinning for HP VM with numa and manual cpu pinning on different hosts
Summary: Incorrect IO and emulator threads auto-pinning for HP VM with numa and manual...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.3.2.1
Hardware: x86_64
OS: Linux
unspecified
medium vote
Target Milestone: ovirt-4.3.5
: ---
Assignee: Michal Skrivanek
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-19 21:42 UTC by Polina
Modified: 2019-03-24 09:40 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-21 08:28:58 UTC
oVirt Team: Virt
pm-rhel: ovirt-4.3+
pm-rhel: blocker+


Attachments (Terms of Use)
logs (deleted)
2019-03-19 21:42 UTC, Polina
no flags Details

Description Polina 2019-03-19 21:42:55 UTC
Created attachment 1545815 [details]
logs

Description of problem: the problem happens in 4.3.2 only when working with rhel8.0 guest. The same scenario works as expected for guest7.6 in the same environment. 


Version-Release number of selected component (if applicable):
vdsm-4.30.11-1.el7ev.x86_64
ovirt-engine-4.3.2.1-0.1.el7.noarch
qemu-guest-agent-2.12.0-63.module+el8+2833+c7d6d092.x86_64

How reproducible: 100% with /etc/redhat-release Red Hat Enterprise Linux release 8.0 Beta (Ootpa)

Steps to Reproduce:
1.Configure HP VM with following parameters:
Total Virtual CPUs = 2 (in System tab)
pinned to host1 supporting numa (Host tab)
Numa node = 1 and pin this vNuma node to pNuma node0
Configure manual cpu pinning 0#1_1#3 (in Resource Allocation tab)

2.Run on host supporting numa - dumpxml is attached


Actual results: 
in /proc/vm_id/status for IO threads and qemu-kvm  the Cpus_allowed_list:      1-11

Expected results:
in /proc/vm_id/status for IO threads and qemu-kvm  the Cpus_allowed_list:      1,3


Additional info:
please look at in qemu.log starting at 2019-03-19 20:50:44.976+0000: starting up
engine.log
2019-03-19 22:50:48,522+02 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-4) [] VM '82d69c9b-a43e-4baa-83b3-04b728c60b83'(golden_env_mixed_virtio_1_0) moved from 'WaitForLaunch' --> 'PoweringUp'

Attached status-io-threads and status-qemu-kvm from host "/proc/{0}/task/*/status".format(vm_pid).

host topology:


[root@lynx22 141475]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10
node 0 size: 16287 MB
node 0 free: 10552 MB
node 1 cpus: 1 3 5 7 9 11
node 1 size: 16384 MB
node 1 free: 11154 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

[root@lynx22 qemu]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                12
On-line CPU(s) list:   0-11
Thread(s) per core:    1
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz
Stepping:              2
CPU MHz:               1900.115
CPU max MHz:           1900.0000
CPU min MHz:           1200.0000
BogoMIPS:              3800.16
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K
NUMA node0 CPU(s):     0,2,4,6,8,10
NUMA node1 CPU(s):     1,3,5,7,9,11

Comment 1 Ryan Barry 2019-03-20 00:56:24 UTC
Deferring a little since EL8 guests are planned to be tech preview

Comment 2 RHEL Product and Program Management 2019-03-20 00:56:26 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 3 Michal Skrivanek 2019-03-20 05:34:37 UTC
Ryan, it’s very 7nlikely it has anything to do with the guest os. 

Polina, so you created one vnuma node and spread it over two physical numa nodes, why?

Comment 4 Polina 2019-03-20 06:34:55 UTC
This is an existent test - part of the regression set from 01/07/18 for HP VMs.

https://polarion.engineering.redhat.com/polarion/redirect/project/RHEVM3/workitem?id=RHEVM-24297 - Verify IO and emulator threads pinning on high-performance VM that has NUMA and CPU pinning on different nodes.

It passed successfully until the last two automation runs in which rhel8.0 quest image is used.

Comment 5 Michal Skrivanek 2019-03-20 07:21:49 UTC
Thanks Polina, makes sense. Note your attached vdsm.log is from wrong time and doesn't contain anything about the VM. Anyway, not required this time...

2019-03-19 22:50:40,563+02 WARN  [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuildUtils] (default task-107) [vms_syncAction_3ba9f970-a030-4f18] No
 IO thread(s) pinning and Emulator thread(s) pinning for High Performance VM golden_env_mixed_virtio_1_0 82d69c9b-a43e-4baa-83b3-04b728c60b83 due to wrong c
onfiguration: IO Threads is not enabled.

Can you please add logs from REST API creating and updating the VM definition? Also the Template it is using if any.

Comment 6 Polina 2019-03-20 09:39:16 UTC
Created attachment 1545962 [details]
logs including rest apis

I'am attaching the engine and vdsm again. please see in engine.log from 2019-03-20 10:47:51,136+02 INFO 

log with rest APIs - ge_compute_he_4.log.debug

2019-03-20 10:47:58,911 - MainThread - vms - DEBUG - PUT request content is --  url:/ovirt-engine/api/vms/82d69c9b-a43e-4baa-83b3-04b728c60b83 body:<vm>
    <cpu>
        <cpu_tune>
            <vcpu_pins>
                <vcpu_pin>
                    <cpu_set>1</cpu_set>
                    <vcpu>0</vcpu>
                </vcpu_pin>
                <vcpu_pin>
                    <cpu_set>3</cpu_set>
                    <vcpu>1</vcpu>
                </vcpu_pin>
            </vcpu_pins>
        </cpu_tune>
        <topology>
            <cores>2</cores>
        </topology>
    </cpu>
    <placement_policy>
        <affinity>pinned</affinity>
        <hosts>
            <host id="c877b028-98f2-4d1e-b3b0-2dcf71473c1a"/>
        </hosts>
    </placement_policy>
    <type>high_performance</type>
</vm>


the template used for creating vm: http://pastebin.test.redhat.com/740548

Get VM after update output: http://pastebin.test.redhat.com/740554

Comment 8 Michal Skrivanek 2019-03-20 10:12:46 UTC
the VM has 0 iothreads, so no pinning anywhere....

The first item in the log about that VM is just listing it and it already doesn't have iothreads so you need to look further into how was the VM created.

Comment 9 Ryan Barry 2019-03-20 11:53:05 UTC
Can you also post the 7.6 template for comparison?

Comment 11 Polina 2019-03-24 09:40:38 UTC
thank you. indeed,  there is a difference between templates I missed (don't know if template 8 is configured with no io_threads by intent, clarifying this with infra ). so, I added this configuration to test.


Note You need to log in before you can comment on or make changes to this bug.