Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510237 - Libvirt should use ovs-vsctl for setting QoS for TAPs plugged into ovs.
Summary: Libvirt should use ovs-vsctl for setting QoS for TAPs plugged into ovs.
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Michal Privoznik
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-07 01:07 UTC by yalzhang@redhat.com
Modified: 2019-02-22 22:10 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)

Description yalzhang@redhat.com 2017-11-07 01:07:33 UTC
Description of problem:
libvirt sets QoS rules and they remain set until qemu takes over the interface. When qemu is initializing it clears out previous settings set by libvirt.

Version-Release number of selected component (if applicable):
libvirt-3.9.0-1.el7.x86_64
qemu-kvm-rhev-2.10.0-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare an inactive domain with bridge type interface connected to ovs
# virsh dumpxml rhel7.4 | grep /interface -B12
    <interface type='bridge'>
      <mac address='52:54:00:f1:87:de'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='736e1c04-1274-4f68-beec-18e1fbd95bfa'/>
      </virtualport>
      <bandwidth>
        <inbound average='200' peak='300' burst='128'/>
        <outbound average='100' peak='200' burst='128'/>
      </bandwidth>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2.# virsh start rhel7.4
Domain rhel7.4 started

# virsh domiflist rhel7.4 
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     ovsbr0     rtl8139     52:54:00:f1:87:de

# tc class show dev vnet0
class htb 1:1 root leaf 2: prio 0 rate 1600Kbit ceil 2400Kbit burst 128Kb cburst 1599b 

# tc filter show dev vnet0 parent ffff:
===> no output

# tc -d qdisc show dev vnet0 
qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 ver 3.17 direct_qlen 1000
qdisc sfq 2: parent 1:1 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 

===> no ingress

3. But when set Qos on the fly, it works
# cat net.xml
<interface type='bridge'>
       <source bridge='ovsbr0'/>
       <virtualport type='openvswitch'>
       </virtualport>
       <bandwidth>
         <inbound average='500' peak='700' burst='256'/>
         <outbound average='300' peak='400' burst='256'/>
       </bandwidth>
     </interface>

# virsh attach-device rhel7.4 net.xml
Device attached successfully

# virsh domiflist rhel7.4
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     ovsbr0     rtl8139     52:54:00:f1:87:de
vnet1      bridge     ovsbr0     rtl8139     52:54:00:7d:3a:cc ===>new added

# tc -d qdisc show dev vnet1
qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 ver 3.17 direct_qlen 1000
qdisc sfq 2: parent 1:1 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 
qdisc ingress ffff: parent ffff:fff1 ---------------- 

# tc -d class show dev vnet1
class htb 1:1 root leaf 2: prio 0 quantum 42 rate 4Mbit ceil 5600Kbit linklayer ethernet burst 256Kb/1 mpu 0b cburst 1598b/1 mpu 0b level 0

# tc -d filter show dev vnet1 parent ffff:
filter protocol all pref 49152 u32 
filter protocol all pref 49152 u32 fh 800: ht divisor 1 
filter protocol all pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 
  match 00000000/00000000 at 0
 police 0x2 rate 2400Kbit burst 256Kb mtu 64Kb action drop overhead 0b linklayer ethernet 
	ref 1 bind 1

Actual results:
In step 2, there is no ingress or filter rule.

Expected results:
In step 2, there should be ingress and filter rule.

Additional info:

Comment 2 jason wang 2017-11-16 02:51:01 UTC
I don't think this is the bug of qemu which runs with unprivileged, don't even have the ability to change tc of TAP.

CC Michal for more ideas.

Comment 3 Michal Privoznik 2017-11-16 09:19:08 UTC
Well, something is clearing out QoS settings on the TAP. Mabe it's the ovs bridge then? I can see the settings applied when I step libvirt. But as soon as it spawns qemu tc settings are gone.

Comment 4 yalzhang@redhat.com 2017-11-16 11:12:28 UTC
I have tried without libvirt,  all the pre-set rules for tap device will be flushed once it attached to the ovs bridge.

And when setting Qos rules after the tap device already attached to the ovs bridge will work.

In the scenario of comment 0, only 1 direction's rule is flushed.

# ip tuntap add  net1 mode tap
# tc qdisc add dev net1 root handle 1: htb
# tc qdisc add dev net1 ingress
# tc qdisc show dev net1
qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 0
qdisc ingress ffff: parent ffff:fff1 ---------------- 

Output for net1:
# tc class add dev net1 parent 1:0 classid 1:1 htb rate 1Mbit
# tc class show dev net1
class htb 1:1 root prio 0 rate 1000Kbit ceil 1000Kbit burst 1600b cburst 1600b 

Ingress for net1:
# tc filter add dev net1 parent ffff: protocol ip prio 50  u32 match ip src 0.0.0.0/0 police rate 2048kbps burst 1m drop flowid :1

# tc filter show dev net1 parent ffff:
filter protocol ip pref 50 u32 
filter protocol ip pref 50 u32 fh 800: ht divisor 1 
filter protocol ip pref 50 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 
  match 00000000/00000000 at 12
 police 0x7 rate 16384Kbit burst 1Mb mtu 2Kb action drop overhead 0b 
ref 1 bind 1

# ovs-vsctl show
a86f46f4-3cd7-4fef-81ce-d5942640e2ce
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "enp1s0"
            Interface "enp1s0"

# ovs-vsctl add-port ovsbr0 net1

# tc class show dev net1
#
# tc filter show dev net1 parent ffff:
#
# tc qdisc show dev net1
#

All rules will be flushed.

Set the rules again, it works

Comment 5 jason wang 2017-11-16 11:59:02 UTC
Ok. Google gives me some interesting discussion on libvirt-user

https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html
http://docs.openvswitch.org/en/latest/howto/qos/

So it looks like OVS has its own QoS mechanism. It looks to me that libvirt should switch to use OVS internal one. Any more ideas Michal?

Thanks

Comment 6 Michal Privoznik 2017-11-16 14:21:25 UTC
(In reply to jason wang from comment #5)
> Ok. Google gives me some interesting discussion on libvirt-user
> 
> https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html
> http://docs.openvswitch.org/en/latest/howto/qos/
> 
> So it looks like OVS has its own QoS mechanism. It looks to me that libvirt
> should switch to use OVS internal one. Any more ideas Michal?
> 
> Thanks

Okay, now it's more clear. Well, we can try but ideally, we would like to tell ovs that the traffic shaping rules are already set, please do not touch them. But for that I need to talk to ovs developers. Switching back to libvirt then.

Comment 8 yipikai7 2018-07-31 13:54:31 UTC
Michal, do you have any updates on this subject ?

Thanks

Comment 9 Michal Privoznik 2018-08-01 07:46:01 UTC
No update here. I have a lot of bugs on my plate. Sorry.

Comment 10 yipikai7 2018-08-01 14:06:22 UTC
Thanks for the reply anyway, otherwise any visibility ?

I ask because we have some (hundreds) VMs running without ingress QoS on there OVS interfaces, thus we have to make a choice here.

Thanks


Note You need to log in before you can comment on or make changes to this bug.