Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366356 - [ovs-dpdk] Prevent other OVS threads to run in the pmd-cpu-mask cores.
Summary: [ovs-dpdk] Prevent other OVS threads to run in the pmd-cpu-mask cores.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: ---
Assignee: Karthik Sundaravel
QA Contact: Maxim Babushkin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-11 18:20 UTC by Flavio Leitner
Modified: 2017-01-10 08:38 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
When using userspace datapath (DPDK), some non-PMD threads run on the same CPU that runs PMD (configured by `pmd-cpu-mask`). This causes the PMD to be preempted which causes latency spikes, drops, etc. With this update, a fix is implemented within the post-install.yaml files available at: https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/network-functions-virtualization-configuration-guide/#ap-ovsdpdk-post-install.
Clone Of:
Environment:
Last Closed: 2017-01-10 08:38:20 UTC


Attachments (Terms of Use)
Attached the post-install.yaml (deleted)
2016-12-09 12:46 UTC, Karthik Sundaravel
no flags Details

Description Flavio Leitner 2016-08-11 18:20:57 UTC
Description of problem:

When using userspace datapath (DPDK), some non-PMD threads run on the same CPU running PMD (configured by pmd-cpu-mask).  That causes the PMD to be preempted which causes latency spikes, drops, etc.

This bug requests the OVS to set the affinity of the other threads to avoid the the cores masked in pmd-cpu-mask parameter. 


Version-Release number of selected component (if applicable):
ovs 2.5

How reproducible:
Always

Steps to Reproduce:
1. configure ovs-dpdk with single queue, single device, with one or more PMDs
2. check with 'ps' that some threads are running in the PMD's core

Actual results:
The PMD thread is preempted by multiple other OVS threads which causes latency spikes and packet loss.

Expected results:
The PMD thread to run alone.

Additional info:
The following script works around the issue:

----8<----
NON_PMD_CPU_RANGE="21-22"
OVS_PID="$(pidof ovs-vswitchd)"
for pid in $(ps -e -T  | grep ${OVS_PID} | grep -v 'pmd' | awk '{ print $2 }')
do
	taskset -p -c ${NON_PMD_CPU_RANGE} ${pid}
done

tuna -t ovs-vswitchd -CP
----8<----

Comment 2 zenghui.shi 2016-08-11 23:34:27 UTC
Is it possible to assign certain cpu core to specific pmd thread ?
this might be useful when we have two dpdk interfaces from different numa nodes, but added to the same ovs bridge.

Comment 3 zenghui.shi 2016-08-12 03:31:09 UTC
(In reply to zenghui.shi from comment #2)
> Is it possible to assign certain cpu core to specific pmd thread ?
> this might be useful when we have two dpdk interfaces from different numa
> nodes, but added to the same ovs bridge.

this question might be confusing, let me rephrase it :
assume we have two NIC interfaces which are located on two numa nodes separately(interface 1 sits on numa node 1, interface 2 sits on numa node 2), set ovs parameters as below:
1) add the two interfaces into the same ovs bridge, add flows to forward bidirectional traffic,
2) 2 cpu cores(one from numa node 1, the other from numa node 2) be used for pmd treads ;
3) use taskset to pin cpu from numa node 1 to pmd thread 1
4) use taskset to pin cpu from numa node 2 to pmd thread 2
question is, how to make sure the pmd thread 1 poll from interface 1 and pmd thread 2 is polling from interface 2?

Comment 5 Franck Baudin 2016-11-14 13:44:40 UTC
Shouldn't we move this BZ in openvswitch-dpdk component?

Comment 6 Flavio Leitner 2016-11-14 15:56:14 UTC
(In reply to Franck Baudin from comment #5)
> Shouldn't we move this BZ in openvswitch-dpdk component?

There is no openvswitch-dpdk component in Fast Datapath.

Comment 7 Franck Baudin 2016-11-14 16:28:07 UTC
will openvswitch-dpdk component be "closed" and all BZs moved to openvswitch?

Comment 8 Kevin Traynor 2016-11-18 20:23:37 UTC
In OVS 2.5 the core that runs general OVS threads must be specified by the user while the PMD's may be. Using -c 0x<bits> and pmd-cpu-mask.

That gives the user the ability to ensure they are not on the same cores. Is that a good enough resolution for this bz for OVS 2.5? 

If not, I think automagically keeping the general threads and pmd-cpu-mask on separate cores should only be done when the user does not specify pmd-cpu-mask. Otherwise, we would be overriding what the user has configured.

OVS 2.6 threading is different and I think it would benefit from this work more.

Comment 10 Flavio Leitner 2016-11-24 17:56:58 UTC
I don't have access to the reproducer environment anymore, but based on what Kevin said and code review, it seems that if -c and pmd-cpu-mask are correct, all non-PMD threads run on cores specified by -c and PMD threads run on cores specified in pmd-cpu-mask, so they should not overlap (maybe that was the mistake before).

That is enough to resolve this bugzilla.
Thanks!

Comment 11 Karthik Sundaravel 2016-12-07 12:45:18 UTC
This shall be addressed in first-boot scripts

Comment 12 Karthik Sundaravel 2016-12-09 12:46:06 UTC
Created attachment 1229955 [details]
Attached the post-install.yaml

Modified the post install scripts

Comment 13 Yariv 2016-12-11 14:09:05 UTC
(In reply to Karthik Sundaravel from comment #12)
> Created attachment 1229955 [details]
> Attached the post-install.yaml
> 
> Modified the post install scripts
Solution suggested:

Additional network-enviroment.yaml parameter needed NeutronDpdkCoreList
Read later on by post-install.yaml

Comment 14 Karthik Sundaravel 2016-12-12 09:03:23 UTC
The parameter NeutronDpdkCoreList is already defined in network-environment.yaml and is used by post-install.yaml now

Comment 15 Maxim Babushkin 2016-12-13 11:04:42 UTC
The post-install.yaml script provided by Karthik and attached to the BZ has been verified.

Comment 17 Karthik Sundaravel 2017-01-10 08:38:20 UTC
Document changes has already been included in the Release Notes at GA. You can find it in the Known Issues section: https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/release-notes/#idm140265780821648.


Note You need to log in before you can comment on or make changes to this bug.