Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1355773 - POWER8: Libvirt getCPUMap() reports threads as offline CPUs
Summary: POWER8: Libvirt getCPUMap() reports threads as offline CPUs
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: ppc64le
OS: Linux
Target Milestone: ---
Assignee: Andrea Bolognani
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2016-07-12 13:59 UTC by Rafael Folco
Modified: 2018-07-18 14:57 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:

Attachments (Terms of Use)

Description Rafael Folco 2016-07-12 13:59:22 UTC
Description of problem:

getCPUMap() returns 20 online cpus. GetInfo()[2] returns 160. 
"Please, Mr. hypervisor, give me a reliable amount of cpus I can use."

Given the hardware thread design in POWER8 (SMT off in the host hypervisor), getCPUMap()returns a list of CPUs which contains the cores in active state and threads as offline cpus.

Applications (as OpenStack) consuming this information gets confused since Libvirt retrieves all physical CPUs (including threads) as the total amount of vcpus that can be used in the system: getInfo()[2]

getInfo()[2] = 160
getCPUMap() returns 20 cpus "usable"

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Libvirt getInfo[2]

Actual results:
cores as online cpus (e.g. 20)

Expected results:
cores * threads_per_core (e.g. 160)

Retrieve total amount of vcpus including threads so the applications that consume it don't get confused and can actually explore all vcpus.

If the main thread (core) is online, the hw threads (siblings) are online too.

Obviously, user would need to use CPU threads accordingly, respecting the characteristics of the platform for pinning and scheduling.

Additional info:

Comment 1 Andrea Bolognani 2016-07-18 13:06:57 UTC
n reply to Rafael Folco from comment #0)
> Description of problem:
> Libvirt reports incomplete NUMA topology from getCapabilities(), which
> prevents applications that consume the topology info to effectively use NUMA
> with the cpu threads. Such applications, as OpenStack, relies in the
> provided NUMA topology to generate Libvirt XMLs when spawning instances.
> This prevents instances to spawn when number of requested vcpus is higher
> than the number of cores in a NUMA node: e.g.: requesting 8 vcpus (2 core, 4
> threads) in one single NUMA node with 5 cpu cores (cannot fit 8 vcpus in 5
> cpu node).
> There is a initial proposal discussed here:
> Steps to Reproduce:
> Spawn an instance with NUMA flavor in OpenStack (which relies on
> getCapabilities)
> Actual results:
> 5 cpus per node (online cpu cores exposed to the host)
>         <cell id='17'>
>  ...
>           <cpus num='5'>
>             <cpu id='80' socket_id='17' core_id='2208' siblings='80'/>
>             <cpu id='88' socket_id='17' core_id='2224' siblings='88'/>
>             <cpu id='96' socket_id='17' core_id='2272' siblings='96'/>
>             <cpu id='104' socket_id='17' core_id='2280' siblings='104'/>
>             <cpu id='112' socket_id='17' core_id='2288' siblings='112'/>
>           </cpus>
> ...
> Expected results:
> FULL NUMA topology including CPU threads (not exposed to the host),
> including siblings info.
>         <cell id='17'>
>  ...
>           <cpus num='40'>
>             <cpu id='80' socket_id='17' core_id='2208' siblings='80'/>
> ...
>             <cpu id='119' socket_id='17' core_id='xxxx' siblings='119'/>
>           </cpus>
> ...

The actual proposal is

and the resulting topology information would actually end
up looking like

  <cell id='17'>
    <cpus num='5'>
      <cpu id='80' thread_capacity='8'
           socket_id='17' core_id='80' siblings='80'/>
      <cpu id='88' thread_capacity='8'
           socket_id='17' core_id='88' siblings='88'/>
      <cpu id='96' thread_capacity='8'
           socket_id='17' core_id='96' siblings='96'/>
      <cpu id='104' thread_capacity='8'
           socket_id='17' core_id='104' siblings='104'/>
      <cpu id='112' thread_capacity='8'
           socket_id='17' core_id='112' siblings='112'/>

The reason we can't just list all CPUs, even thought we could
infer all required information by looking at the host, is
that there is an existing expectation that you will be able to
manage pCPUs listed there in a certain way, eg. by pinning
vCPUs to them; however, you can't pin vCPUs to secondary host
threads, as that would result in an error.

Would the proposed augmented output still satisfy your use
case? Please note that I've glossed over some details here, but
the mailing list message linked above contains more information.

Comment 2 Rafael Folco 2016-07-19 13:51:15 UTC
The bug against NUMA Topology is actually this one:
Moving last comment over there.

Comment 3 Andrea Bolognani 2016-07-19 14:08:09 UTC
Yup, I had both bugs open in the browser and I mistakenly
replied to the wrong one. Quite embarassing! Thanks for
cleaning up after my mess :)

Comment 4 Rafael Folco 2016-07-19 14:23:11 UTC
Np. With the proposed solution for NUMA topology in, I'd assume getCPUMap() will remain reporting threads as offline cpus. Correct assumption?
Thank you.

Note You need to log in before you can comment on or make changes to this bug.