Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1356937 - libvit should support set IOthread quota into cgroup
Summary: libvit should support set IOthread quota into cgroup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: John Ferlan
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-15 10:22 UTC by Pei Zhang
Modified: 2016-11-03 18:48 UTC (History)
7 users (show)

Fixed In Version: libvirt-2.0.0-8.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 18:48:59 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Description Pei Zhang 2016-07-15 10:22:12 UTC
Description of problem:
Currently, we do not support set iothread quota. But it would make a failure when we want to set global quota in some situations.

Version-Release number of selected component (if applicable):
libvirt-2.0.0-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. start a guest with configured vcpu quota.
# virsh start vm1
Domain vm1 started

# virsh schedinfo vm1
Scheduler      : posix
cpu_shares     : 2048
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

2. Add an iothread to guest. 
# virsh iothreadadd vm1 1 

check in cgroup, it has iothread1
# ls /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d23\\x2dvm1.scope/iothread1/
.....

3. Change the vcpu_quota 

# virsh schedinfo vm1 --set vcpu_quota=1000 
Scheduler      : posix
cpu_shares     : 2048
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

4. change global_quota, it will fail to change.

# virsh schedinfo vm1 --set global_quota=2000 
Scheduler      : posix
error: Invalid value '2000' for 'cpu.cfs_quota_us': Invalid argument

5. check value in iothread 
# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d23\\x2dvm1.scope/iothread1/cpu.cfs_quota_us 
4000

6. change the value in cgroup, then set global value again. 
# echo "1000" > /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d23\\x2dvm1.scope/iothread1/cpu.cfs_quota_us 

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d23\\x2dvm1.scope/iothread1/cpu.cfs_quota_us 
1000

# virsh schedinfo vm1 --set global_quota=2000 
Scheduler      : posix
cpu_shares     : 2048
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000


Actual results:
As steps 4, it will fail to set global quota.

Expected results:
We cannot set iothread quota now, but it will block us for setting global quota in this situation.
So we should support set iothread quota. 

Additional info:

Comment 2 John Ferlan 2016-07-25 21:48:21 UTC
Without any changes, using a build of the same libvirtd referenced by this bz, I'm not seeing the same results using your example steps. 

Here's my example using your steps, but slightly expanded (there's just one domain that will be running, hence use of machine-qemu* and iothread*):

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory

# virsh start dom
Domain  started

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory

# virsh schedinfo dom
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

# virsh dumpxml dom | grep iothr
# virsh iothreadadd dom 1
# virsh dumpxml dom | grep iothr
  <iothreads>1</iothreads>
  <iothreadids>
    <iothread id='1'/>
  </iothreadids>

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
# virsh schedinfo dom --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# virsh schedinfo dom --set global_quota=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1

...

So as you can see by this point I can set global quota and you'll also note that there is a -1 in my *iothread*/cpu.cfs_quota_us file. Since yours shows 4000, I'm wondering if by chance you "altered" the *iothread1/cpu.cfs_quota_us file to 4000 *before* you tried modifying the global...

It seems so, if I modify the file "by hand" to a value *lower* than what I'm about to set to the global value, then I get the same failure you note when trying to set to 2000 (since the IOThread is at 4000). If instead of 2000 I use 5000, then the global setting works. That is:

# echo "4000" > /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
# virsh schedinfo dom --set vcpu_quota=1000
...works
# virsh schedinfo dom --set global_quota=2000
Scheduler      : posix
error: Invalid value '2000' for 'cpu.cfs_quota_us': Invalid argument
# virsh schedinfo f18 --set global_quota=5000
...works


So in a way, your example is slightly flawed. Since global setting was only added in libvirt 1.3.3 (e.g. after RHEL 7.2), there isn't any sort of regression here.  So I think this bz may need to move to rhel-7.4. Although I will try for 7.3 - may not happen since there's a change to a header file required (libvirt-domain.h).


Regardless of where it falls, I have put together some patches that would allow one to modify the the "iothread_quota" and "iothread_period" much in the same way it is done for vcpu, emulator, and global values.  This at least then prints out whatever value that IOThreads are using and allows setting for IOThreads specifically.

Used a different domain with 4 IOThreads already defined... The number doesn't matter, just that they're defined. Here's the results:

# virsh start iothrdom
# virsh schedinfo iothrdom
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
-1
-1
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1
# virsh schedinfo iothrdom --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1
# virsh schedinfo iothrdom --set global_quota=2000 
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1

# virsh schedinfo iothrdom --set global_quota=2500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : -1

# virsh schedinfo iothrdom --set iothread_quota=1500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1500

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
1500
1500
1500
1500

The patches are posted upstream as:

http://www.redhat.com/archives/libvir-list/2016-July/msg01024.html

Comment 3 John Ferlan 2016-08-03 12:03:50 UTC
Upstream change made:

commit e4e4d17c9c2828b3a7d2909295e724c57e60a334
Author: John Ferlan <jferlan@redhat.com>
Date:   Mon Jul 25 07:07:43 2016 -0400

    qemu: Add support to get/set IOThread period and quota cgroup values
    
...
    
    Add support for IOThread quota/bandwidth and period parameters for non
    session mode. If in session mode, then error out.  Uses all the same
    places where {vcpu|emulator|global}_{period|quota} are adjusted and
    adds the iothread values.

Comment 7 yisun 2016-09-13 10:22:17 UTC
Document check with libvirt-docs-2.0.0-8.el7.x86_64:

     </dd><dt><code>iothread_period</code></dt><dd>
        The optional <code>iothread_period</code> element specifies the
        enforcement interval(unit: microseconds) for IOThreads. Within
        <code>iothread_period</code>, each IOThread of the domain will
        not be allowed to consume more than <code>iothread_quota</code>
        worth of runtime. The value should be in range [1000, 1000000].
        An iothread_period with value 0 means no value.
        <span class="since">Only QEMU driver support since 2.1.0</span>
      </dd><dt><code>iothread_quota</code></dt><dd>
        The optional <code>iothread_quota</code> element specifies the maximum
        allowed bandwidth(unit: microseconds) for IOThreads. A domain with
        <code>iothread_quota</code> as any negative value indicates that the
        domain IOThreads have infinite bandwidth, which means that it is
        not bandwidth controlled. The value should be in range
        [1000, 18446744073709551] or less than 0. An <code>iothread_quota</code>
        with value 0 means no value. You can use this feature to ensure that
        all IOThreads run at the same speed.
        <span class="since">Only QEMU driver support since 2.1.0</span>

Comment 8 yisun 2016-09-13 10:31:40 UTC
Scenario 1: iothread_quota test

# virsh schedinfo virtlab_test_copy
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1


# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
4000
4000
4000
4000



# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory


# virsh iothreadadd virtlab_test_copy 1
# virsh iothreadadd virtlab_test_copy 2


# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1

# virsh schedinfo virtlab_test_copy --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1


# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
1000
1000

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1


# virsh schedinfo virtlab_test_copy --set global_quota=2000 
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000
iothread_period: 100000
iothread_quota : -1

#  cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
1000
1000

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1


# virsh schedinfo virtlab_test_copy --set global_quota=500
Scheduler      : posix
error: invalid argument: value of 'global_quota' is out of range [1000, 18446744073709551]


# virsh schedinfo virtlab_test_copy --set global_quota=2500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : -1


# virsh schedinfo virtlab_test_copy --set iothread_quota=1500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1500


# virsh schedinfo virtlab_test_copy --set iothread_quota=1300
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1300


# virsh schedinfo virtlab_test_copy --set iothread_quota=2600
Scheduler      : posix
error: Invalid value '2600' for 'cpu.cfs_quota_us': Invalid argument
<==== should not larger than 2500, as expected

# virsh schedinfo virtlab_test_copy --set global_quota=1200
Scheduler      : posix
error: Invalid value '1200' for 'cpu.cfs_quota_us': Invalid argument
<==== should not less than 1300, as expected

# virsh dumpxml virtlab_test_copy | grep thread_quota -i
    <iothread_quota>1300</iothread_quota>

Comment 9 yisun 2016-09-13 10:40:07 UTC
Hi John, 
for iothread_period, I don't quite get the logic
Is it working with global_period?

In my test, I have two iothreads, 4 vcpus and global_period is 100000, as follow:
# virsh dumpxml virtlab_test_copy | egrep "iothread|vcpu"
  <vcpu placement='static'>4</vcpu>
  <iothreads>2</iothreads>
  <iothreadids>
    <iothread id='2'/>
    <iothread id='1'/>
  </iothreadids>
    <iothread_period>51999</iothread_period>
    <iothread_quota>1300</iothread_quota>

And I can set iothread_period to 51999, but not 51998, what's the logic?

[root@localhost ~]# virsh schedinfo virtlab_test_copy --set iothread_period=51999
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 51999
iothread_quota : 1300

[root@localhost ~]# virsh schedinfo virtlab_test_copy --set iothread_period=51998
Scheduler      : posix
error: Invalid value '51998' for 'cpu.cfs_period_us': Invalid argument

Comment 10 John Ferlan 2016-09-14 10:45:50 UTC
Not quite sure. The error comes from *deep* in the bowels of the cgroup logic - when a virFileWriteStr() fails. That would mean the "EINVAL" (Invalid argument) was returned from either an open(), write(), or close() command. Neither makes sense since the previous adjustment worked. 

In the long run the iothread_period logic is essentially a copy of vcpu_period and emulator_period logic, so do those fail similarly? Did you try another update after that (e.g. back to 51999 or 60000 or 1000)?  How much more debugging could be done to see where that error comes from? It'd probably be useful to get some sort of syscall tracking to know which was called last (open, write, close).

Of course attempting a similar sequence on my upstream libvirt doesn't have the same issue.

Comment 11 yisun 2016-09-20 06:55:39 UTC
Verified on: 
libvirt-2.0.0-9.el7.x86_64
kernel-3.10.0-508.el7.x86_64


# virsh start virtlab_test_copy
Domain virtlab_test_copy started

# virsh iothreadadd virtlab_test_copy 1

#  virsh schedinfo virtlab_test_copy
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# virsh schedinfo virtlab_test_copy --set iothread_period=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 2000
iothread_quota : -1

# virsh schedinfo virtlab_test_copy --set iothread_quota=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 2000
iothread_quota : 2000

# virsh dumpxml virtlab_test_copy | grep "iothread"
  <iothreads>1</iothreads>
  <iothreadids>
    <iothread id='1'/>
  </iothreadids>
    <iothread_period>2000</iothread_period>
    <iothread_quota>2000</iothread_quota>

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d4\\x2dvirtlabtestcopy.scope/iothread1/cpu.cfs_quota_us 
2000

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d4\\x2dvirtlabtestcopy.scope/iothread1/cpu.cfs_period_us 
2000


=======
the question mentioned in comment 9 is now doced in a new cgroup bug 1377574.

Comment 13 errata-xmlrpc 2016-11-03 18:48:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html


Note You need to log in before you can comment on or make changes to this bug.