Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1514088 - DPDK support for bnxt PMD
Summary: DPDK support for bnxt PMD
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dpdk
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: beta
: 7.5
Assignee: Davide Caratti
QA Contact: Jean-Tsung Hsiao
URL:
Whiteboard:
Depends On: 1471943
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-16 15:59 UTC by Andy Gospodarek
Modified: 2018-08-22 13:52 UTC (History)
33 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1471943
Environment:
Last Closed: 2018-08-20 13:21:20 UTC


Attachments (Terms of Use)
ovsdb-server.log-20171206.gz (deleted)
2017-12-06 15:08 UTC, Dilip Daya
no flags Details
ovs-vswitchd.log-20171206.gz (deleted)
2017-12-06 15:09 UTC, Dilip Daya
no flags Details
openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm (deleted)
2017-12-06 17:44 UTC, Dilip Daya
no flags Details
cmd log to get OVS working with bnxt PMD (deleted)
2017-12-07 06:14 UTC, Somnath Kotur
no flags Details
openvswitch-2.6.1-16.git20161206.el7ost.src.rpm (deleted)
2017-12-07 08:53 UTC, Vinayak
no flags Details
dpdk-16.11-3.el7fdp.src.rpm (deleted)
2017-12-07 08:56 UTC, Vinayak
no flags Details
tarball of dpdk-16.11(extracted from dpdk-16.11-3.el7fdp.src.rpm) (deleted)
2017-12-08 04:55 UTC, Somnath Kotur
no flags Details
tarball of openvswitch-2.6.1( extracted from openvswitch-2.6.1-16.git20161206.el7ost.src.rpm) (deleted)
2017-12-08 05:01 UTC, Somnath Kotur
no flags Details
dpdk-16.11-3.el7.src rpm with patch for bnxt applied (deleted)
2017-12-08 10:07 UTC, Somnath Kotur
no flags Details
rhel-7-fast-datapath-htb-rpms_failure_report (deleted)
2018-01-19 17:37 UTC, Dilip Daya
no flags Details

Description Andy Gospodarek 2017-11-16 15:59:13 UTC
+++ This bug was initially created as a clone of Bug #1471943 +++
+++ This bug was initially created as a clone of Bug #1470370 +++

DPDK has supported the bnxt PMD since 16.07.

Filing this to be sure that the OVS+DPDK included in RHEL OSP10 has support for the bnxt PMD.

Comment 1 Ajit Khaparde 2017-11-16 19:55:02 UTC
What version of DPDK is being planned for inclusion?
Please let us know if Broadcom needs to provide anything from our side?

Thanks

Comment 2 Sean Merrow 2017-11-21 14:30:38 UTC
The HPE 631 NIC, which uses the Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller, is not currently supported by Red Hat for DPDK.  This is because DPDK testing has not been completed and proven to work, and therefore the PMD for bnxt has been disabled.

Red Hat DOES ship the driver with the OVS-DPDK bits as part of RHEL/OSP, but it is simply disabled in a file called openvswitch.spec

cat openvswitch.spec
  :
# Disable some PMDs on fdProd
setconf CONFIG_RTE_LIBRTE_BNXT_PMD n   <-- 631 PMD (Broadcom)

This request is to have support in RHEL 7 so it can be used in an Red Hat OpenStack Platform (OSP) deployment for NFV

OSP 10 - OVS 2.6 w/DPDK 16.11
OSP 11 - OVS 2.6 w/DPDK 16.11
OSP 12 - OVS 2.2 w/DPDK 16.11 (beta is out today, GA targeted for Dec. 2017)

HPE is looking for this to work in OSP 10, which relies on the bnxt PMD to be enabled in RHEL 7.

Comment 3 Sean Merrow 2017-11-21 15:04:22 UTC
Correction to above note. OSP 12 line should say:

OSP 12 - OVS 2.7 w/DPDK 16.11 (beta is out today, GA targeted for Dec. 2017)

Also, note that there is no upstream work to do here. The PMD for bnxt is enabled upstream, but only disabled downstream in Red Hat product.

Comment 4 Jakub Libosvar 2017-11-22 10:12:48 UTC
Sean,

can you please clarify what is expected from Neutron team to work on in order to make this request completed?

Comment 5 Sean Merrow 2017-11-22 13:15:30 UTC
This is not related to Neutron, so resetting the product and component. Also adding Broadcom EPM to CC list.

Broadcom can reach out to their EPM for guidance on how to get the DPDK PMD enabled and supported in RHEL 7.

Comment 10 Dilip Daya 2017-12-01 14:17:11 UTC
Environment:
* ProLiant DL380 Gen10
* RHEL7.4 (3.10.0-693.5.2.el7.x86_64)
* NIC: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
  - Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28 Adapter

+ /home/dilip/dpdk-17.11/usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=

# lspci -ns 0000:12:00.0
12:00.0 0200: 14e4:16d7


vfio-pci using dpdk + ovs matrix 
--------------------------------

works         = dpdk ports using vfio-pci are identified
OVS fails     = Error attaching device to DPDK
testpmd fails = EAL: No probed ethernet devices
not-tested    = testpmd fails I don't expect ovs to work

                        testpmd         upstream OVS                            RH-OVS
upstream dpdk-17.11     works           compiler failure: upstream ovs-2.8.1    fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
upstream dpdk-17.08     works           works: upstream ovs-2.8.1               fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
dpdk-17.05-2.el7        fails           not-tested                              fails: openvswitch-2.7.0-8.git20170530.el7.x86_64

rhel-7-fast-datapath-htb-rpms based on:
- openvswitch-2.7.0-8
- dpdk-17.05-2


Debug details for OVS
---------------------

=> RH: openvswitch-2.7.0-8.git20170530

# Start the ovs-vswitchd:
++ ovs-ctl --no-ovsdb-server --db-sock=/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
                                 [  OK  ]
Enabling remote OVSDB managers   [  OK  ]
...


=> compared to working upstream ovs:

# Start the ovs-vswitchd:
++ /home/dilip/openvswitch-2.8.1/utilities/ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd Zone 0: name:<rte_eth_dev_data>, phys:0x83ffb6dc0, len:0x34000, virt:0x7f317ffb6dc0, socket_id:0, flags:0
Zone 1: name:<bnxt_0000:12:00:00-rx_port_stat>, phys:0x83ffb6300, len:0x440, virt:0x7f317ffb6300, socket_id:0, flags:0
Zone 2: name:<bnxt_0000:12:00:00-tx_port_stat>, phys:0x83ffb5f00, len:0x3c0, virt:0x7f317ffb5f00, socket_id:0, flags:0
Zone 3: name:<bnxt_0000:12:00:00-0000_def_cp>, phys:0x83ffb4000, len:0x1000, virt:0x7f317ffb4000, socket_id:0, flags:0
Zone 4: name:<bnxt_0000:12:00:00_vnicattr>, phys:0x83ff8e1c0, len:0x1be40, virt:0x7f317ff8e1c0, socket_id:0, flags:0
Zone 5: name:<bnxt_0000:12:00:01-rx_port_stat>, phys:0x83ff82fc0, len:0x440, virt:0x7f317ff82fc0, socket_id:0, flags:0
Zone 6: name:<bnxt_0000:12:00:01-tx_port_stat>, phys:0x83ff82bc0, len:0x3c0, virt:0x7f317ff82bc0, socket_id:0, flags:0
Zone 7: name:<bnxt_0000:12:00:01-0000_def_cp>, phys:0x83ff81000, len:0x1000, virt:0x7f317ff81000, socket_id:0, flags:0
Zone 8: name:<bnxt_0000:12:00:01_vnicattr>, phys:0x83ff5af80, len:0x1c000, virt:0x7f317ff5af80, socket_id:0, flags:0
                                  [  OK  ]
Enabling remote OVSDB managers    [  OK  ]
...

Comment 11 Andy Gospodarek 2017-12-01 19:24:48 UTC
I tried for a while yesterday afternoon and was not able to make the latest tip of tree DPDK work with VFIO.  I have not bisected to figure out if this was in-fact a regression.  I'm hopeful that the development team will weigh-in shortly.

Comment 12 Rashid Khan 2017-12-01 20:51:51 UTC
Thanks for the info Andy. 
Please keep us posted. We are getting a lot of pressure from the field about this.

Comment 13 Somnath Kotur 2017-12-04 03:33:43 UTC
Hi
   Not sure i understand all of the issues cited in this bug, but one thing i was able to figure out is that 16.11 code base is missing support for the device stated. It was not there in 16.11 including LTS

This is the patch that they are missing and this line in particular
(+#define BROADCOM_DEV_ID_57414 0x16d7)

commit: 1f4a84672ecbd28bfaaa77f99b6e5b9b77896415
net/bnxt: support new PCI IDs
Add support for PCI IDs which was removed as a part of commit
0e7bd0b2d6a5 ("net/bnxt: remove support for few devices")
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

That patch(manually applied to 16.11) and the ovs spec file change to include BNXT PMD should be able to get us going with OSP 10 ? 

Thanks
Som

Comment 14 Somnath Kotur 2017-12-04 09:56:40 UTC
BTW Andy, we were able to get basic DPDK load work with VFIO with latest tip of tree (http://dpdk.org/git/next/dpdk-next-net)

Thanks
Som

Comment 15 Andy Gospodarek 2017-12-04 20:04:16 UTC
Thanks, Somanth!  It seems like HPE's concern is with 16.11.  I just tested the latest and didn't get bnxt with VFIO to come up on my system.  I was able to get VFIO working with another PMD.

Comment 16 Somnath Kotur 2017-12-05 02:20:12 UTC
Hi Andy,
 So do we know if HPE folks tried applying the patch i mentioned in comment#13 on to their 16.11 code base and give it a shot?  I don't think there will be another 16.11 release, so they would just have to apply this patch on their sandbox manually along with the change to the OVS Spec file to include BNXT PMD i guess?

Thanks
Som

Comment 17 Sandeep Raman 2017-12-05 06:53:21 UTC
Hi Som,

Re #Comment 10, 

                        testpmd         upstream OVS                            RH-OVS
upstream dpdk-17.11     works           compiler failure: upstream ovs-2.8.1    fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
upstream dpdk-17.08     works           works: upstream ovs-2.8.1               fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
dpdk-17.05-2.el7        fails           not-tested                              fails: openvswitch-2.7.0-8.git20170530.el7.x86_64


dpdk-17.05-2.el7 testpmd fails. But dpdk-17.05-2.el7 includes both the device id and the Y flag for bnxt. Was testpmd not supposed to work in this scenario?

grep -i 0x16d7 SOURCES/dpdk-17.05/drivers/net/bnxt/bnxt_ethdev.c
#define BROADCOM_DEV_ID_57414 0x16d7

Regards,
Sandeep.

Comment 18 Andy Gospodarek 2017-12-06 04:54:55 UTC
(In reply to Somnath Kotur from comment #16)
> Hi Andy,
>  So do we know if HPE folks tried applying the patch i mentioned in
> comment#13 on to their 16.11 code base and give it a shot?  I don't think
> there will be another 16.11 release, so they would just have to apply this
> patch on their sandbox manually along with the change to the OVS Spec file
> to include BNXT PMD i guess?
> 
> Thanks
> Som

I do not know.  HPE can you confirm if you have tried Somnath's suggestion?

Comment 19 Dilip Daya 2017-12-06 05:14:36 UTC
Hi Andy,

RPMS provided by Sandeep Raman:
- dpdk-16.11-3.el7.x86_64.rpm
- dpdk-16.11-3.el7.src.rpm

[root@en104 ~]# uname -r
3.10.0-693.11.1.el7.x86_64

[root@en104 ~]# rpm -qa | grep dpdk
dpdk-16.11-3.el7.x86_64
dpdk-tools-16.11-3.el7.x86_64


[root@en104 ~]# lsmod | grep vfio
[root@en104 ~]#
[root@en104 ~]# modprobe vfio-pci
[root@en104 ~]# lsmod | grep vfio
vfio_pci               41267  0
vfio_iommu_type1       22300  0
vfio                   32367  2 vfio_iommu_type1,vfio_pci
irqbypass              13503  2 kvm,vfio_pci


[root@en104 ~]# dpdk-devbind --bind vfio-pci 0000:12:00.0 0000:12:00.1
[root@en104 ~]# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


[root@en104 ~]# P0=0000:12:00.0
[root@en104 ~]# P1=0000:12:00.1
[root@en104 ~]# /usr/bin/testpmd -c 0xe -n 4 -w $P0 -w $P1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
Interactive-mode selected
EAL: Error - exiting with code: 1
  Cause: Invalid port 2


Removed any deps on environment variables:

[root@en104 ~]# /usr/bin/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
Interactive-mode selected
EAL: Error - exiting with code: 1
  Cause: Invalid port 2




Using dpdk.org--upstream dpdk-17.08 on the same system:
-------------------------------------------------------

Removed dpdk-16.11-3 rpms:
[root@en104 dpdk-17.08]# rpm -qa | grep dpdk
[root@en104 dpdk-17.08]#


[root@en104 dpdk-17.08]# usertools/dpdk-devbind.py --bind vfio-pci 0000:12:00.0 0000:12:00.1
[root@en104 dpdk-17.08]# usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=

[root@en104 dpdk-17.08]# ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt

PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.7.7
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7fcbc0000000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.7.7
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7fcbc0112000M
Interactive-mode selected
Set io packet forwarding mode
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1
Configuring Port 0 (socket 0)
PMD: Port 0 Link Up - speed 25000 Mbps - full-duplex
PMD: VLAN Filtering: 1
PMD: VLAN Strip Offload: 1
Port 0: 9C:DC:71:B3:B8:A0
Configuring Port 1 (socket 0)
PMD: Port 1 Link Up - speed 25000 Mbps - full-duplex
PMD: VLAN Filtering: 1
PMD: VLAN Strip Offload: 1
Port 1: 9C:DC:71:B3:B8:A8
Checking link statuses...
Done
testpmd>

Comment 20 Somnath Kotur 2017-12-06 05:55:02 UTC
(In reply to Sandeep Raman from comment #17)
> Hi Som,
> 
> Re #Comment 10, 
> 
>                         testpmd         upstream OVS                        
> RH-OVS
> upstream dpdk-17.11     works           compiler failure: upstream ovs-2.8.1
> fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> upstream dpdk-17.08     works           works: upstream ovs-2.8.1           
> fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> dpdk-17.05-2.el7        fails           not-tested                          
> fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> 
> 
> dpdk-17.05-2.el7 testpmd fails. But dpdk-17.05-2.el7 includes both the
> device id and the Y flag for bnxt. Was testpmd not supposed to work in this
> scenario?
> 
> grep -i 0x16d7 SOURCES/dpdk-17.05/drivers/net/bnxt/bnxt_ethdev.c
> #define BROADCOM_DEV_ID_57414 0x16d7
> 
> Regards,
> Sandeep.

Hi Sandeep,
          OK i am a bit confused here re: the actual requirement/issue at hand here. I thought (like Andy) that the real issue is getting BNXT PMD to work with DPDK 16.11, is that no longer the case now? 
Seems like the goalpost has shifted ? 
1. Do we now want this to work with dpdk-17.05-2.el7?
2. Did we try applying the patch i suggested in comment#13 to 16.11 code base and see if it worked? 
I tried patching 16.11.3 (LTS) with the missing patch that i had mentioned earlier in my comment# 13 and with that testpmd loads fine on the mentioned card even on 16.11 with VFIO PCI.
I also tried with 17.05-2 release and that seemed to work fine as well without any patch.
3. Could you please share the logs(console output) when testpmd was loaded so we can see the failure on 17.05-2 ?
4. Please share the FW version on your card? 
Load L2 driver (bnxt_en) and give us the output of 'ethtool -i ethX' (interface corresponding to the PCIe function that you want DPDK to run on)

Thanks
Som

Comment 21 Sandeep Raman 2017-12-06 06:12:20 UTC
(In reply to Somnath Kotur from comment #20)
> (In reply to Sandeep Raman from comment #17)
> > Hi Som,
> > 
> > Re #Comment 10, 
> > 
> >                         testpmd         upstream OVS                        
> > RH-OVS
> > upstream dpdk-17.11     works           compiler failure: upstream ovs-2.8.1
> > fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> > upstream dpdk-17.08     works           works: upstream ovs-2.8.1           
> > fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> > dpdk-17.05-2.el7        fails           not-tested                          
> > fails: openvswitch-2.7.0-8.git20170530.el7.x86_64
> > 
> > 
> > dpdk-17.05-2.el7 testpmd fails. But dpdk-17.05-2.el7 includes both the
> > device id and the Y flag for bnxt. Was testpmd not supposed to work in this
> > scenario?
> > 
> > grep -i 0x16d7 SOURCES/dpdk-17.05/drivers/net/bnxt/bnxt_ethdev.c
> > #define BROADCOM_DEV_ID_57414 0x16d7
> > 
> > Regards,
> > Sandeep.
> 
> Hi Sandeep,
>  OK i am a bit confused here re: the actual requirement/issue at
> hand here. I thought (like Andy) that the real issue is getting BNXT PMD to work with DPDK 16.11, is that no longer the case now?
This is still the case.
 
> Seems like the goalpost has shifted ? No.

> 1. Do we now want this to work with dpdk-17.05-2.el7? - HPE has been exploring different combinations which may help in better understanding what works and what doesn't. Since dpdk-17.05-2.el7 had support for BCM631, was surprised to see testpmd fail. Hence checked if you know why the failure.

> 2. Did we try applying the patch i suggested in comment#13 to 16.11 code
> base and see if it worked? 
Yes we applied the patch suggested in comment#13 to 16.11 code base but it did not work and failed with the same footprint as observed with dpdk-17.05-2.el7 [See #Comment19]

> I tried patching 16.11.3 (LTS) with the missing patch that i had mentioned
earlier in my comment# 13 and with that testpmd loads fine on the mentioned
card even on 16.11 with VFIO PCI.


> I also tried with 17.05-2 release and that seemed to work fine as well
> without any patch.
> 3. Could you please share the logs(console output) when testpmd was loaded
> so we can see the failure on 17.05-2 ?
> 4. Please share the FW version on your card? 
> Load L2 driver (bnxt_en) and give us the output of 'ethtool -i ethX'
> (interface corresponding to the PCIe function that you want DPDK to run on)

Dilip, could you please provide the data Som has asked, Thanks Sandeep.
> 
> Thanks
> Som

Comment 22 Dilip Daya 2017-12-06 07:20:21 UTC
Hi Sandeep / Som:

As requested in Comment 21 see below:


# lspci -ns 0000:12:00.0
12:00.0 0200: 14e4:16d7

[root@en104 ~]# ethtool -i ens2f0
driver: bnxt_en
version: 1.7.0
firmware-version: 20.6.135/1.8.0 pkg 20.06.0601
expansion-rom-version: 
bus-info: 0000:12:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no


[root@en104 ~]# lspci -vvs 0000:12:00.0
12:00.0 Ethernet controller: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
	Subsystem: Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28 Adapter
	Physical Slot: 2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 26
	NUMA node: 0
	Region 0: Memory at da600000 (64-bit, prefetchable) [size=64K]
	Region 2: Memory at da400000 (64-bit, prefetchable) [size=1M]
	Region 4: Memory at da620000 (64-bit, prefetchable) [size=8K]
	[virtual] Expansion ROM at da680000 [disabled] [size=512K]
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data
		Product Name: HPE Eth 10/25Gb 2p 631SFP28 Adptr
		Read-only fields:
			[PN] Part number: 817716-001
			[EC] Engineering changes: A-5716
			[MN] Manufacture ID: 31 35 39 30
			[V0] Vendor specific: SU 25G/10G GEN3 x8 14W
			[V1] Vendor specific: 20.6.119
			[V3] Vendor specific: 20.6.135
			[V5] Vendor specific: 0A
			[V6] Vendor specific: 20.6.101
			[V7] Vendor specific: 631SFP28
			[VA] Vendor specific: 20.06.0601 
			[SN] Serial number: 1CH7190075
			[V2] Vendor specific: 5719
			[V4] Vendor specific: 9CDC71B3B8A0
			[RV] Reserved: checksum good, 164 byte(s) reserved
		End
	Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00001000
	Capabilities: [ac] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <1us, L1 <2us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE#
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [13c v1] Device Serial Number 9c-dc-71-ff-fe-b3-b8-a0
	Capabilities: [150 v1] Power Budgeting <?>
	Capabilities: [160 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
	Capabilities: [1b0 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 16, stride: 1, Device ID: 16c1
		Supported Page Size: 000005ff, System Page Size: 00000001
		Region 0: Memory at 00000c7ffff60000 (64-bit, prefetchable)
		Region 2: Memory at 00000c7fff700000 (64-bit, prefetchable)
		Region 4: Memory at 00000c7ffff40000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [230 v1] Transaction Processing Hints
		Interrupt vector mode supported
		Device specific mode supported
		Steering table in MSI-X table
	Capabilities: [300 v1] #19
	Capabilities: [200 v1] Precision Time Measurement
		PTMCap: Requester:+ Responder:- Root:-
		PTMClockGranularity: Unimplemented
		PTMControl: Enabled:- RootSelected:-
		PTMEffectiveGranularity: Unknown
	Kernel driver in use: bnxt_en
	Kernel modules: bnxt_en


[root@en104 x86_64]# rpm -qa | grep dpdk
dpdk-17.05-2.el7.x86_64
dpdk-tools-17.05-2.el7.x86_64


[root@en104 x86_64]# dpdk-devbind --bind vfio-pci 0000:12:00.0 0000:12:00.1
[root@en104 x86_64]# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=



[root@en104 x86_64]# /usr/bin/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7fbe1be00000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7fbe1bf12000M
Interactive-mode selected
Set io packet forwarding mode
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1
Configuring Port 0 (socket 0)
PMD: Port 0 Link Up - speed 25000 Mbps - full-duplex
PMD: MAC addr already existed for pool 0
Port 0: 9C:DC:71:B3:B8:A0
Configuring Port 1 (socket 0)
PMD: Port 1 Link Up - speed 25000 Mbps - full-duplex
PMD: MAC addr already existed for pool 0
Port 1: 9C:DC:71:B3:B8:A8
Checking link statuses...
Done
testpmd> 


# pwd
/home/dilip/RH_OVS-DPDK_POC_jason/rhel-7-fast-datapath_packages
[root@en104 rhel-7-fast-datapath_packages]# l
total 31044
-rw-r--r-- 1 root root  1582708 Nov 30 06:12 dpdk-17.05-2.el7fdb.x86_64.rpm
-rw-r--r-- 1 root root  4921120 Nov 30 06:12 openvswitch-2.7.0-8.git20170530.el7fdb.x86_64.rpm
-rw-r--r-- 1 root root 15335503 Nov 30 06:12 openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
-rw-r--r-- 1 root root  9937208 Nov 30 06:12 dpdk-17.05-2.el7fdb.src.rpm


[root@en104 x86_64]# rpm -Uvh openvswitch-2.7.0-8.git20170530.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:openvswitch-1:2.7.0-8.git20170530################################# [100%]
[root@en104 x86_64]# rpm -qa | grep openv
openvswitch-2.7.0-8.git20170530.el7.x86_64


[root@en104 ovs-2.8.1_startup_scripts]# ./start_ovs_and_looped_flows_no_comments.sh 
++ export DB_SOCK=/var/run/openvswitch/db.sock
++ DB_SOCK=/var/run/openvswitch/db.sock
++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/share/openvswitch/scripts/:/var/run/
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/share/openvswitch/scripts/:/var/run/
++ export OVSSCHEMA=/usr/share/openvswitch/vswitch.ovsschema
++ OVSSCHEMA=/usr/share/openvswitch/vswitch.ovsschema
++ P1=0000:12:00.0
++ P2=0000:12:00.1
++ ovs-ctl stop
ovs-vswitchd is not running.
ovsdb-server is not running.
++ pkill -9 ovs
++ rm -rf /var/run/openvswitch
++ rm -rf /etc/openvswitch/
++ rm -rf /etc/openvswitch/conf.db
++ rm -fr /var/log/openvswitch/
++ mkdir -p /etc/openvswitch
++ mkdir -p /var/run/openvswitch
++ mkdir -p /var/log/openvswitch
++ modprobe openvswitch
++ ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
++ sleep 6
++ ovs-ctl --system-id=random --no-ovs-vswitchd start
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x6
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
++ sleep 6
++ ovs-ctl --no-ovsdb-server --db-sock=/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
++ sleep 6
++ ovs-ctl status
ovsdb-server is running with pid 14801
ovs-vswitchd is running with pid 14854
++ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport0 -- set Interface dpdkport0 type=dpdk options:dpdk-devargs=0000:12:00.0 ofport_request=1
ovs-vsctl: Error detected while setting up 'dpdkport0': Error attaching device '0000:12:00.0' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport1 -- set Interface dpdkport1 type=dpdk options:dpdk-devargs=0000:12:00.1 ofport_request=2
ovs-vsctl: Error detected while setting up 'dpdkport1': Error attaching device '0000:12:00.1' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
++ sleep 8
++ ovs-vsctl set Interface dpdkport0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
++ sleep 5
++ ovs-vsctl set Interface dpdkport1 options:n_rxq=1 other_config:pmd-rxq-affinity=0:2
++ sleep 5
++ ovs-vsctl set Interface dpdkport0 options:n_rxq_desc=2048
++ ovs-vsctl set Interface dpdkport1 options:n_rxq_desc=2048
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=34.101s, table=0, n_packets=0, n_bytes=0, idle_age=34, priority=0 actions=NORMAL
++ ovs-ofctl del-flows br0
++ sleep 5
++ ovs-ofctl add-flow br0 in_port=1,action=output:2
++ ovs-ofctl add-flow br0 in_port=2,action=output:1
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=2 actions=output:1
++ ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000e66a22130f44
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 LOCAL(br0): addr:e6:6a:22:13:0f:44
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
++ ovs-appctl dpif-netdev/pmd-stats-show
main thread:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:595108 (100.00%)
	processing cycles:0 (0.00%)
++ ovs-appctl dpif-netdev/pmd-rxq-show
++ ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 1 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
++ ovs-vsctl list interface
_uuid               : 9b2fa3aa-9a62-40c1-a690-b3bdb5ce8957
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "Error attaching device '0000:12:00.0' to DPDK"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport0"
ofport              : -1
ofport_request      : 1
options             : {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {}
status              : {}
type                : dpdk

_uuid               : f1ee0a4e-0978-47bf-911a-2ecd858a70c7
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "Error attaching device '0000:12:00.1' to DPDK"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport1"
ofport              : -1
ofport_request      : 2
options             : {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:2"}
statistics          : {}
status              : {}
type                : dpdk

_uuid               : 1f814887-8c9b-413a-a0f9-ba23bc4d0179
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 17
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "e6:6a:22:13:0f:44"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal
++ ps -ef
++ grep ovs
root     14800     1  0 00:14 ?        00:00:00 ovsdb-server: monitoring pid 14801 (healthy)
root     14801 14800  0 00:14 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root     14853     1  0 00:14 ?        00:00:00 ovs-vswitchd: monitoring pid 14854 (healthy)
root     14854 14853 47 00:14 ?        00:00:40 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
root     14929 14752  0 00:16 pts/0    00:00:00 grep ovs
[root@en104 ovs-2.8.1_startup_scripts]#

Comment 23 Somnath Kotur 2017-12-06 10:16:44 UTC
Hi Dilip,
          Interesting to note that you don't seem to have any problem loading testpmd with dpdk-17.05-2 on your card which is in sync with what we are seeing in our labs as well.  
I was given to understand that testpmd load itself is failing, which going by above is not. It seems to be failing further down during the OVS setup.

Are you sure you have 'quit' the testpmd that was loaded/running, that could 
be one reason why the attach failed ?

Also, could you please share the ovs-vswitchd.log (/var/log/openvswitch) or /var/log/messages - wherever you have configured and can see on your setup)?

Meanwhile, we will try to repro it locally on our setup here

Thanks
Som

Comment 24 Somnath Kotur 2017-12-06 11:55:47 UTC
Dilip,

 The other thing is you seem to be using *el7.rpms for both DPDK and OVS while we are using tarballs available freely on the website.

Could you or Andy help us get access to these rpms?

Thanks
Som

Comment 25 Dilip Daya 2017-12-06 14:52:43 UTC
(In reply to Somnath Kotur from comment #24)
> Dilip,
> 
>  The other thing is you seem to be using *el7.rpms for both DPDK and OVS
> while we are using tarballs available freely on the website.
> 
> Could you or Andy help us get access to these rpms?

The *el7.rpms were provided by Sandeep Raman.

Thanks,
-DilipD.

Comment 26 Dilip Daya 2017-12-06 15:07:26 UTC
(In reply to Somnath Kotur from comment #23)
> Hi Dilip,
>           Interesting to note that you don't seem to have any problem
> loading testpmd with dpdk-17.05-2 on your card which is in sync with what we
> are seeing in our labs as well.  
> I was given to understand that testpmd load itself is failing, which going
> by above is not.

Previously I may have leftover remnants of upstream bits on my system
and made sure I cleaned up the upstream version as well as rebuilt the
rpms from src rpms for the latest installed kernel.


> It seems to be failing further down during the OVS setup.

Correct:   OVS fails     = Error attaching device to DPDK

 
> Are you sure you have 'quit' the testpmd that was loaded/running, that could 
> be one reason why the attach failed ?

Maybe.

 
> Also, could you please share the ovs-vswitchd.log (/var/log/openvswitch) or
> /var/log/messages - wherever you have configured and can see on your setup)?


Attached:
- ovsdb-server.log-20171206.gz
- ovs-vswitchd.log-20171206.gz


> Meanwhile, we will try to repro it locally on our setup here

Thanks,
-DilipD.

Comment 27 Dilip Daya 2017-12-06 15:08:57 UTC
Created attachment 1363708 [details]
ovsdb-server.log-20171206.gz

Comment 28 Dilip Daya 2017-12-06 15:09:41 UTC
Created attachment 1363709 [details]
ovs-vswitchd.log-20171206.gz

Comment 29 Andy Gospodarek 2017-12-06 15:36:24 UTC
(In reply to Dilip Daya from comment #25)
> (In reply to Somnath Kotur from comment #24)
> > Dilip,
> > 
> >  The other thing is you seem to be using *el7.rpms for both DPDK and OVS
> > while we are using tarballs available freely on the website.
> > 
> > Could you or Andy help us get access to these rpms?
> 
> The *el7.rpms were provided by Sandeep Raman.
> 
> Thanks,
> -DilipD.

Provided where?  They are not attached to this bugzilla and I do not see any in my inbox.  Sandeep, can you please attach them to this bugzilla?

Thanks!

Comment 30 Sandeep Raman 2017-12-06 16:52:14 UTC
(In reply to Andy Gospodarek from comment #29)
> (In reply to Dilip Daya from comment #25)
> > (In reply to Somnath Kotur from comment #24)
> > > Dilip,
> > > 
> > >  The other thing is you seem to be using *el7.rpms for both DPDK and OVS
> > > while we are using tarballs available freely on the website.
> > > 
> > > Could you or Andy help us get access to these rpms?
> > 
> > The *el7.rpms were provided by Sandeep Raman.
> > 
> > Thanks,
> > -DilipD.
> 
> Provided where?  They are not attached to this bugzilla and I do not see any
> in my inbox.  Sandeep, can you please attach them to this bugzilla?

RH provided link to download rpm's
https://access.redhat.com/downloads/content/rhel---7/x86_64/5423/dpdk/16.11-3.el7fdp/x86_64/fd431d51/package
https://access.redhat.com/downloads/content/rhel---7/x86_64/5153/dpdk/17.05-2.el7fdb/x86_64/fd431d51/package
> 
> Thanks!

Comment 31 Dilip Daya 2017-12-06 17:44:55 UTC
Created attachment 1363799 [details]
openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm

Comment 32 Somnath Kotur 2017-12-07 02:46:28 UTC
Thanks Dilip for the rpms! 

BTW did you try unloading /quitting the testpmd and then loading OVS ? 
Did it work?

Thanks
Som

Comment 33 Dilip Daya 2017-12-07 04:27:20 UTC
(In reply to Somnath Kotur from comment #32)
> Thanks Dilip for the rpms! 
> 
> BTW did you try unloading /quitting the testpmd and then loading OVS ? 
> Did it work?


=> REBOOT SYSTEM:
[root@en104 ovs-2.8.1_startup_scripts]# uptime
 21:22:36 up 2 min,  1 user,  load average: 0.56, 0.49, 0.21

=> Load DPDK:
[root@en104 ovs-2.8.1_startup_scripts]# ./load_dpdk_nics.sh 
+ export BINDER=dpdk-devbind
+ BINDER=dpdk-devbind
+ ulimit -l unlimited
+ P1=0000:12:00.0
+ P2=0000:12:00.1
+ INTERFACE_1=ens2f0
+ INTERFACE_2=ens2f1
+ ip link set ens2f0 up
+ ip link set ens2f1 up
Cannot find device "ens2f1"
+ chmod a+x /dev/vfio
+ chmod 0666 /dev/vfio/vfio
+ modprobe vfio
+ sleep 3
+ modprobe vfio-pci
+ sleep 3
+ dpdk-devbind --bind vfio-pci 0000:12:00.0 0000:12:00.1
+ sleep 5
+ dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=


[root@en104 ovs-2.8.1_startup_scripts]# lsmod | grep vfio
vfio_pci               41267  0 
vfio_iommu_type1       22300  0 
vfio                   32367  2 vfio_iommu_type1,vfio_pci
irqbypass              13503  2 kvm,vfio_pci

=> Load OVS:
[root@en104 ovs-2.8.1_startup_scripts]# ./start_ovs_and_looped_flows_no_comments.sh 
++ export DB_SOCK=/var/run/openvswitch/db.sock
++ DB_SOCK=/var/run/openvswitch/db.sock
++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/share/openvswitch/scripts/:/var/run/
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/share/openvswitch/scripts/:/var/run/
++ export OVSSCHEMA=/usr/share/openvswitch/vswitch.ovsschema
++ OVSSCHEMA=/usr/share/openvswitch/vswitch.ovsschema
++ P1=0000:12:00.0
++ P2=0000:12:00.1
++ ovs-ctl stop
ovs-vswitchd is not running.
ovsdb-server is not running.
++ pkill -9 ovs
++ rm -rf /var/run/openvswitch
++ rm -rf /etc/openvswitch/
++ rm -rf /etc/openvswitch/conf.db
++ rm -fr /var/log/openvswitch/
++ mkdir -p /etc/openvswitch
++ mkdir -p /var/run/openvswitch
++ mkdir -p /var/log/openvswitch
++ modprobe openvswitch
++ ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
++ sleep 6
++ ovs-ctl --system-id=random --no-ovs-vswitchd start
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x6
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
++ sleep 6
++ ovs-ctl --no-ovsdb-server --db-sock=/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
++ sleep 6
++ ovs-ctl status
ovsdb-server is running with pid 3500
ovs-vswitchd is running with pid 3582
++ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport0 -- set Interface dpdkport0 type=dpdk options:dpdk-devargs=0000:12:00.0 ofport_request=1
ovs-vsctl: Error detected while setting up 'dpdkport0': Error attaching device '0000:12:00.0' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport1 -- set Interface dpdkport1 type=dpdk options:dpdk-devargs=0000:12:00.1 ofport_request=2
ovs-vsctl: Error detected while setting up 'dpdkport1': Error attaching device '0000:12:00.1' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
++ sleep 8
++ ovs-vsctl set Interface dpdkport0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
++ sleep 5
++ ovs-vsctl set Interface dpdkport1 options:n_rxq=1 other_config:pmd-rxq-affinity=0:2
++ sleep 5
++ ovs-vsctl set Interface dpdkport0 options:n_rxq_desc=2048
++ ovs-vsctl set Interface dpdkport1 options:n_rxq_desc=2048
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=34.068s, table=0, n_packets=0, n_bytes=0, idle_age=34, priority=0 actions=NORMAL
++ ovs-ofctl del-flows br0
++ sleep 5
++ ovs-ofctl add-flow br0 in_port=1,action=output:2
++ ovs-ofctl add-flow br0 in_port=2,action=output:1
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=2 actions=output:1
++ ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000caec2b1fb84c
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 LOCAL(br0): addr:ca:ec:2b:1f:b8:4c
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
++ ovs-appctl dpif-netdev/pmd-stats-show
main thread:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:629742 (100.00%)
	processing cycles:0 (0.00%)
++ ovs-appctl dpif-netdev/pmd-rxq-show
++ ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 1 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
++ ovs-vsctl list interface
_uuid               : bd7b3e01-cf61-4aef-9a7e-ac0c6d3bd318
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 13
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "ca:ec:2b:1f:b8:4c"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal

_uuid               : a380fce2-69d9-48b0-b5da-2e47a2b5317d
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "Error attaching device '0000:12:00.1' to DPDK"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport1"
ofport              : -1
ofport_request      : 2
options             : {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:2"}
statistics          : {}
status              : {}
type                : dpdk

_uuid               : a853df27-00a3-4b0b-8a98-0edc8e716b2c
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "Error attaching device '0000:12:00.0' to DPDK"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport0"
ofport              : -1
ofport_request      : 1
options             : {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {}
status              : {}
type                : dpdk
++ ps -ef
++ grep ovs
root      3499     1  0 21:23 ?        00:00:00 ovsdb-server: monitoring pid 3500 (healthy)
root      3500  3499  0 21:23 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root      3581     1  0 21:24 ?        00:00:00 ovs-vswitchd: monitoring pid 3582 (healthy)
root      3582  3581 46 21:24 ?        00:00:39 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
root      3775  3443  0 21:25 pts/0    00:00:00 grep ovs
[root@en104 ovs-2.8.1_startup_scripts]# 
[root@en104 ovs-2.8.1_startup_scripts]# 
[root@en104 ovs-2.8.1_startup_scripts]# 
[root@en104 ovs-2.8.1_startup_scripts]# cat /var/log/openvswitch/ovs
ovsdb-server.log  ovs-vswitchd.log  
[root@en104 ovs-2.8.1_startup_scripts]# cat /var/log/openvswitch/ovsdb-server.log 
2017-12-07T04:23:28.967Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
2017-12-07T04:23:28.971Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.7.0
2017-12-07T04:23:38.985Z|00003|memory|INFO|2664 kB peak resident set size after 10.0 seconds
2017-12-07T04:23:38.985Z|00004|memory|INFO|cells:16 monitors:0
[root@en104 ovs-2.8.1_startup_scripts]# 
[root@en104 ovs-2.8.1_startup_scripts]# cat /var/log/openvswitch/ovs-vswitchd.log 
2017-12-07T04:24:05.090Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2017-12-07T04:24:05.094Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0
2017-12-07T04:24:05.094Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1
2017-12-07T04:24:05.094Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores
2017-12-07T04:24:05.094Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-12-07T04:24:05.094Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2017-12-07T04:24:05.095Z|00007|dpdk|INFO|DPDK Enabled - initializing...
2017-12-07T04:24:05.095Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2017-12-07T04:24:05.095Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x1 --socket-mem 2048,0
2017-12-07T04:24:45.007Z|00010|dpdk|INFO|DPDK pdump packet capture enabled
2017-12-07T04:24:45.013Z|00011|dpdk|INFO|DPDK Enabled - initialized
2017-12-07T04:24:45.014Z|00012|timeval|WARN|Unreasonably long 39919ms poll interval (31ms user, 39692ms system)
2017-12-07T04:24:45.014Z|00013|timeval|WARN|faults: 82601 minor, 0 major
2017-12-07T04:24:45.014Z|00014|timeval|WARN|disk: 0 reads, 8 writes
2017-12-07T04:24:45.014Z|00015|timeval|WARN|context switches: 1 voluntary, 42 involuntary
2017-12-07T04:24:45.014Z|00016|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=fde89638:
2017-12-07T04:24:45.014Z|00017|coverage|INFO|bridge_reconfigure         0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00018|coverage|INFO|cmap_expand                0.6/sec     0.150/sec        0.0025/sec   total: 9
2017-12-07T04:24:45.014Z|00019|coverage|INFO|miniflow_malloc            0.6/sec     0.150/sec        0.0025/sec   total: 9
2017-12-07T04:24:45.014Z|00020|coverage|INFO|hmap_pathological          0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00021|coverage|INFO|hmap_expand               11.2/sec     6.333/sec        0.1056/sec   total: 380
2017-12-07T04:24:45.014Z|00022|coverage|INFO|txn_unchanged              0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-07T04:24:45.014Z|00023|coverage|INFO|txn_incomplete             0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00024|coverage|INFO|poll_create_node           2.0/sec     0.667/sec        0.0111/sec   total: 40
2017-12-07T04:24:45.014Z|00025|coverage|INFO|seq_change                 1.6/sec     0.733/sec        0.0122/sec   total: 44
2017-12-07T04:24:45.014Z|00026|coverage|INFO|pstream_open               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00027|coverage|INFO|stream_open                0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00028|coverage|INFO|util_xalloc              209.2/sec   121.433/sec        2.0239/sec   total: 7286
2017-12-07T04:24:45.014Z|00029|coverage|INFO|netdev_get_hwaddr          0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-07T04:24:45.014Z|00030|coverage|INFO|netlink_received           0.6/sec     0.050/sec        0.0008/sec   total: 3
2017-12-07T04:24:45.014Z|00031|coverage|INFO|netlink_sent               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-07T04:24:45.014Z|00032|coverage|INFO|86 events never hit
2017-12-07T04:24:45.016Z|00033|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (<->/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (99% CPU usage)
2017-12-07T04:24:45.016Z|00034|memory|INFO|66128 kB peak resident set size after 39.9 seconds
2017-12-07T04:24:45.016Z|00035|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.7.0
2017-12-07T04:24:51.129Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-12-07T04:24:51.129Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1
2017-12-07T04:24:51.129Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-12-07T04:24:51.129Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action
2017-12-07T04:24:51.129Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-12-07T04:24:51.129Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state
2017-12-07T04:24:51.129Z|00042|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone
2017-12-07T04:24:51.129Z|00043|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark
2017-12-07T04:24:51.129Z|00044|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label
2017-12-07T04:24:51.129Z|00045|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state_nat
2017-12-07T04:24:51.138Z|00046|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-07T04:24:51.138Z|00047|bridge|INFO|bridge br0: using datapath ID 0000caec2b1fb84c
2017-12-07T04:24:51.138Z|00048|connmgr|INFO|br0: added service controller "punix:/var/run/openvswitch/br0.mgmt"
2017-12-07T04:24:59.148Z|00049|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:24:59.148Z|00050|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:07.157Z|00051|netdev_dpdk|WARN|Error attaching device '0000:12:00.1' to DPDK
2017-12-07T04:25:07.158Z|00052|netdev|WARN|dpdkport1: could not set configuration (Invalid argument)
2017-12-07T04:25:07.158Z|00053|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:25:07.158Z|00054|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:15.163Z|00055|netdev_dpdk|WARN|Error attaching device '0000:12:00.1' to DPDK
2017-12-07T04:25:15.163Z|00056|netdev|WARN|dpdkport1: could not set configuration (Invalid argument)
2017-12-07T04:25:15.163Z|00057|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:25:15.163Z|00058|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:20.169Z|00059|netdev_dpdk|WARN|Error attaching device '0000:12:00.1' to DPDK
2017-12-07T04:25:20.169Z|00060|netdev|WARN|dpdkport1: could not set configuration (Invalid argument)
2017-12-07T04:25:20.169Z|00061|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:25:20.169Z|00062|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:25.174Z|00063|netdev_dpdk|WARN|Error attaching device '0000:12:00.1' to DPDK
2017-12-07T04:25:25.174Z|00064|netdev|WARN|dpdkport1: could not set configuration (Invalid argument)
2017-12-07T04:25:25.175Z|00065|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:25:25.175Z|00066|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:25.178Z|00067|netdev_dpdk|WARN|Error attaching device '0000:12:00.1' to DPDK
2017-12-07T04:25:25.178Z|00068|netdev|WARN|dpdkport1: could not set configuration (Invalid argument)
2017-12-07T04:25:25.178Z|00069|netdev_dpdk|WARN|Error attaching device '0000:12:00.0' to DPDK
2017-12-07T04:25:25.178Z|00070|netdev|WARN|dpdkport0: could not set configuration (Invalid argument)
2017-12-07T04:25:25.209Z|00071|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 deletes)
2017-12-07T04:25:30.211Z|00072|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-12-07T04:25:30.214Z|00073|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
[root@en104 ovs-2.8.1_startup_scripts]#

Comment 34 Somnath Kotur 2017-12-07 06:12:37 UTC
Hi Dilip,
          So i was able to get it working on my setup with the OVS rpm provided by you and dpdk-16.11.3 (http://fast.dpdk.org/rel/dpdk-16.11.3.tar.xz) 
With this OVS version, we need DPDK-16.11.3 otherwise OVS compilation fails with error : 'rte_virtio_net.h' file not found (This file seems to be available only in DPDK-16.11.3).
Note: I patched my dpdk-16.11.3 codebase with the patch i had specified earlier in comment#13.

Below are the sequence of cmds issued on my setup, will attach it as a file as well to the JIRA.
I just followed the steps mentioned in the Documentation/intro/install/dpdk.rst and Documentation/intro/install/general.rst inside the OVS folder itself.


[root@dhcp-10-123-156-144 rhel7_fast_data_path_packages]# rpm -ivh openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
Updating / installing...
   1:openvswitch-1:2.7.0-8.git20170530################################# [100%]
[root@dhcp-10-123-156-144 rhel7_fast_data_path_packages]# cd /root/rpmbuild/SPECS
[root@dhcp-10-123-156-144 SPECS]# rpmbuild -bp openvswitch.spec
[root@dhcp-10-123-156-144 SPECS]# cd ../BUILD
[root@dhcp-10-123-156-144 BUILD]# cd openvswitch-2.7.0/
[root@dhcp-10-123-156-144 openvswitch-2.7.0] cd /root/dpdk-stable-16.11.3/
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3] export DPDK_DIR=`pwd`
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3] export DPDK_TARGET=x86_64-native-linuxapp-gcc
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3] export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3]  make install T=$DPDK_TARGET DESTDIR=install
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3]  $DPDK_DIR/tools/dpdk-devbind.py --bind=vfio-pci 0000:05:00.0
[root@dhcp-10-123-156-144 dpdk-stable-16.11.3]  $DPDK_DIR/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en

[root@dhcp-10-123-156-144 dpdk-stable-16.11.3]  cd -
[root@dhcp-10-123-156-144 openvswitch-2.7.0]
[root@dhcp-10-123-156-144 openvswitch-2.7.0] ./configure --with-dpdk=$DPDK_BUILD
[root@dhcp-10-123-156-144 openvswitch-2.7.0] make
[root@dhcp-10-123-156-144 openvswitch-2.7.0] make install
[root@dhcp-10-123-156-144 openvswitch-2.7.0] make modules_install
[root@dhcp-10-123-156-144 openvswitch-2.7.0] modprobe openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# lsmod | grep openv
openvswitch           106775  0
nf_nat_ipv6            14131  1 openvswitch
nf_defrag_ipv6         35104  2 openvswitch,nf_conntrack_ipv6
libcrc32c              12644  1 openvswitch
nf_nat_ipv4            14115  2 openvswitch,iptable_nat
nf_nat                 26147  4 openvswitch,nf_nat_ipv4,nf_nat_ipv6,nf_nat_masquerade_ipv4
nf_conntrack          111302  8 openvswitch,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_conntrack_ipv6

[root@dhcp-10-123-156-144 openvswitch-2.7.0] export PATH=$PATH:/usr/local/share/openvswitch/scripts
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# grep HugePages_ /proc/meminfo
HugePages_Total:    2048
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0

[root@dhcp-10-123-156-144 openvswitch-2.7.0] export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# rm -rf /usr/local/var/run/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# mkdir -p /usr/local/var/run/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file
2017-12-07T04:41:17Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovsdb-server.log
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl --no-wait init
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
Starting ovs-vswitchd EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: printing PMD version: bnxt_hwrm_ver_get
PMD: 1.8.0:20.8.13
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem d60d0000, node addr 0x7fe9b9e00000M
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x66a1ced40, len:0x30100, virt:0x7fe99cfced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:05:00.0 ofport_request=1
(reverse-i-search)`add-port': ovs-vsctl ^Cd-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:05:00.0 ofport_request=1
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl list-ports br0
dpdk0
[root@dhcp-10-123-156-144 openvswitch-2.7.0]#  ps -eLo pid,psr,comm | grep pmd
26752   0 pmd21
root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:05:00.0 ofport_request=1 ^C
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl set Interface dpdk0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl set Interface dpdk0 options:n_rxq_desc=2048
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=544.213s, table=0, n_packets=0, n_bytes=0, idle_age=544, priority=0 actions=NORMAL
[root@dhcp-10-123-156-144 openvswitch-2.7.0]# ovs-vsctl list interface
_uuid               : 8434fa88-105d-46a8-b958-4fca6e526c7a
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 16
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "00:0a:f9:ca:6b:10"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal

_uuid               : b05326a9-727d-4c8d-ae85-8b1d75426df5
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : []
external_ids        : {}
ifindex             : 0
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : []
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "00:0a:f9:ca:6b:10"
mtu                 : 1500
mtu_request         : []
name                : "dpdk0"
ofport              : 1
ofport_request      : 1
options             : {dpdk-devargs="0000:05:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=net_bnxt, max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="242", max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1", numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="0"}
type                : dpdk

Comment 35 Somnath Kotur 2017-12-07 06:14:26 UTC
Created attachment 1364049 [details]
cmd log to get OVS working with bnxt PMD

Comment 36 Dilip Daya 2017-12-07 07:08:00 UTC
Unfortunately I cannot respond until next week as I've been on vacation all this week. As of today I will not have internet access. Back in on Dec 13th.

Comment 37 Vinayak 2017-12-07 08:50:43 UTC
Hi Somnath,

Does this mean that the original RHOSP 10 bundled DPDK (dpdk-16.11-3.el7fdp.src.rpm) and OVS-DPDK (openvswitch-2.6.1-16.git20161206.el7ost.src.rpm) should work with the patch you have mentioned in Comment#13? The OVS-DPDK RPM has http://fast.dpdk.org/rel/dpdk-16.11.3.tar.xz embedded in it.

Can you please help to validate with these RPMs in your sandbox - I will share these files.

Regards,
Vinayak

Comment 38 Vinayak 2017-12-07 08:53:59 UTC
Created attachment 1364113 [details]
openvswitch-2.6.1-16.git20161206.el7ost.src.rpm

This is the OVS+DPDK RPM packaged with RHOSP10.

Comment 39 Vinayak 2017-12-07 08:56:40 UTC
Created attachment 1364115 [details]
dpdk-16.11-3.el7fdp.src.rpm

And this is the DPDK library source - dpdk-16.11-3.el7fdp.src.rpm in RHOSP10.

Comment 40 Vinayak 2017-12-07 09:03:37 UTC
Hi Som,

Can you please help validate Comment#37?

Regards,
Vinayak

Comment 41 Somnath Kotur 2017-12-07 09:33:51 UTC
(In reply to Vinayak from comment #37)
> Hi Somnath,
> 
> Does this mean that the original RHOSP 10 bundled DPDK
> (dpdk-16.11-3.el7fdp.src.rpm) and OVS-DPDK
> (openvswitch-2.6.1-16.git20161206.el7ost.src.rpm) should work with the patch
> you have mentioned in Comment#13? The OVS-DPDK RPM has
> http://fast.dpdk.org/rel/dpdk-16.11.3.tar.xz embedded in it.
> 
Yes, that's what i had already suggested and had asked you to validate since the same had worked on my sandbox.
> Can you please help to validate with these RPMs in your sandbox - I will
> share these files.
Well, almost all the versions of OVS(2.7, 2.8, latest upstream OVS)/DPDK(16.11.3 with my patch, 17.05-2, 17.08, 17.11) you have suggested have worked on my sandbox, we need to see why it's not working on your sandbox and how to get it working which is why i'd provided the steps i did in comment# 34.
In any case, providing the steps for OVS-2.6.1 (RPM you had provided above) on my sandbox

[root@dhcp-10-123-156-144 openvswitch-2.6.1]# export PATH=$PATH:/usr/local/share/openvswitch/scripts
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# rm -rf /usr/local/etc/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# mkdir -p /usr/local/etc/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# rm -rf /usr/local/var/run/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# mkdir -p /usr/local/var/run/openvswitch
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file
2017-12-07T09:25:49Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovsdb-server.log
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl --no-wait init
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
Starting ovs-vswitchd EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: printing PMD version: bnxt_hwrm_ver_get
PMD: 1.8.0:20.8.13
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem d60d0000, node addr 0x7fbab7200000M
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x66a1ced40, len:0x30100, virt:0x7fba9a3ced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
[root@dhcp-10-123-156-144 openvswitch-2.6.1]#  ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:05:00.0 ofport_request=1
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl list-ports br0
dpdk0
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl set Interface dpdk0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl set Interface dpdk0 options:n_rxq_desc=2048
[root@dhcp-10-123-156-144 openvswitch-2.6.1]# ovs-vsctl list interface
_uuid               : 14d89301-acc9-4b5d-8512-fde8fa026c2d
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : []
external_ids        : {}
ifindex             : 0
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : []
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "00:0a:f9:ca:6b:10"
mtu                 : 1500
mtu_request         : []
name                : "dpdk0"
ofport              : 1
ofport_request      : 1
options             : {dpdk-devargs="0000:05:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=net_bnxt, max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="242", max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1", numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="0"}
type                : dpdk

_uuid               : baf8a5ea-5348-4de0-84ec-a05f5483b8ed
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 18
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "00:0a:f9:ca:6b:10"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal
[root@dhcp-10-123-156-144 openvswitch-2.6.1]#


Thanks
Som

Comment 42 Vinayak 2017-12-07 09:57:58 UTC
Thank you Som. 

Just to confirm (did not find it in your BZ update), you rebuilt the OVS+DPDK/ DPDK package using the source RPM I had shared after making the changes you mentioned in Comment#13 before running the test?

We will re-check what is different in our sandbox in Dilip's lab - (through whoever is standing-in in his absence).

Regards,
Vinayak

Comment 43 Somnath Kotur 2017-12-07 10:22:50 UTC
(In reply to Vinayak from comment #42)
> Thank you Som. 
> 
> Just to confirm (did not find it in your BZ update), you rebuilt the
> OVS+DPDK/ DPDK package using the source RPM I had shared after making the
> changes you mentioned in Comment#13 before running the test?
Well your comment below said that it is the same as the one on dpdk.org/rel which i already have installed on my sandbox.
<snip>
The OVS-DPDK RPM has http://fast.dpdk.org/rel/dpdk-16.11.3.tar.xz embedded in it.
</snip>

> We will re-check what is different in our sandbox in Dilip's lab - (through
> whoever is standing-in in his absence).
yes ,please do and revert

Thanks
Som

Comment 44 Dilip Daya 2017-12-07 22:31:27 UTC
Hi Som,

I used the following src rpms:
- openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
- dpdk-16.11-3.el7fdp.src.rpm

1. Made changes to both SPEC files for dpdk and ovs:

"setconf CONFIG_RTE_LIBRTE_BNXT_PMD y"


2. Applied patch commit from:
<http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415> to both dpdk-16.11-3.el7fdp.src.rpm 
and dpdk tree within openvswitch-2.6.1-16.git20161206.el7ost.src.rpm

3. Built both dpdk and ovs and installed:

[root@en104 ~]# rpm -qa | grep dpdk
dpdk-tools-16.11-3.el7.x86_64
dpdk-16.11-3.el7.x86_64

[root@en104 ~]# rpm -qa | grep openv
openvswitch-2.6.1-16.git20161206.el7.x86_64

4. Test dpdk and OVS:

+ dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


# /usr/bin/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
Interactive-mode selected
EAL: Error - exiting with code: 1
  Cause: Invalid port 2

as well as OVS.

5. Requesting Som to provide me dpdk and ovs src rpms that have all the code changes applied. All I then have to do is install the sources and "rpmbuild -ba" to create the RPMS and install to test.


Note: I do not guarantee that I will be able to get to this this week--I'll try my best.  Worst case: test on Dec-13th.

Thanks,
-DilipD.

Comment 45 Somnath Kotur 2017-12-08 04:55:31 UTC
Created attachment 1364622 [details]
tarball of dpdk-16.11(extracted from dpdk-16.11-3.el7fdp.src.rpm)

Comment 46 Somnath Kotur 2017-12-08 05:01:02 UTC
Created attachment 1364623 [details]
tarball of openvswitch-2.6.1( extracted from openvswitch-2.6.1-16.git20161206.el7ost.src.rpm)

Comment 47 Somnath Kotur 2017-12-08 05:06:19 UTC
Dilip,
      For now i have attached the compressed tarballs itself, you just need to extract them and run the steps i have listed in comment# 34. 
There is no change in the OVS tarball per se, just building the DPDK with the steps i have listed should suffice.
Again, this is some setup /configuration related issue as even with this dpdk (extracted from the rpm) which seems to be slightly different from (http://fast.dpdk.org/rel/dpdk-16.11.3.tar.xz) ( just one file - bnxt_hwrm.c, but that doesn't seem to matter) and my patch applied, i was able to get it to work.

So please retry with them and revert. If the folks here in India/BLR cannot get access to your setup, maybe you might have to just work with Ajit (in your timezone) on your setup to see what's going on?

Thanks
Som

Comment 48 Somnath Kotur 2017-12-08 06:01:46 UTC
Dilip,
 Just to elaborate more, please follow these steps after extracting the DPDK tarball that i provided above

1) rpm -ivh <ovs.rpm> ( you can use the same  ovs-2.6.1.src.rpm that you have/attached to this bug)
2) cd /root/rpmbuild/SPECS
3) rpmbuild -bp openvswitch.spec(NO change) (you might need to 'yum install' some other pkgs to get this working)
4) cd ../BUILD/openvswitch-2.6.1
5) ./configure --with-dpdk=$DPDK_BUILD (set DPDK_BUILD and other DPDK_* env variables to the extracted folder from the tarball that i provided)
6) make (Type `autoreconf -f -i` if it complains about aclocal-1.14 command 
         not found on your system)
7) make install
8) make modules_install
9) modprobe openvswitch
10) export PATH=$PATH:/usr/local/share/openvswitch/scripts
11) ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
12) rm -rf /usr/local/var/run/openvswitch
13) mkdir -p /usr/local/var/run/openvswitch
14) ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file
15) export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
16) ovs-vsctl --no-wait init
17) ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
18) ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
19) ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
20) ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:0b:00.0 ofport_request=1
21) ovs-vsctl list interface

Comment 49 Somnath Kotur 2017-12-08 10:07:04 UTC
Created attachment 1364724 [details]
dpdk-16.11-3.el7.src rpm with patch for bnxt applied

Comment 50 Somnath Kotur 2017-12-08 10:14:36 UTC
Dilip,
 Just in case, i have even provided the src rpm (with the patch http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 for bnxt PMD applied) 

TO make things simple and to ensure things work, please first install this rpm using below steps:

1) rpm -ivh '<src.rpm>' 
2) cd /root/rpmbuild/SOURCES/
3) tar -xvf dpdk-16.11.tar.xz
4) cd dpdk-16.11/
5) export DPDK_DIR=`pwd`
6) export DPDK_TARGET=x86_64-native-linuxapp-gcc
7) export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
8) make install T=$DPDK_TARGET DESTDIR=install

These 8 steps should get the DPDK libraries/apps built including bnxt PMD. 

Post these, please run the steps to configure OVS as per comment# 48.

Thanks
Som

Comment 51 Dilip Daya 2017-12-08 10:50:17 UTC
(In reply to Somnath Kotur from comment #50)
> Dilip,
>  Just in case, i have even provided the src rpm (with the patch
> http://www.dpdk.org/browse/dpdk/commit/
> ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 for bnxt PMD applied) 
> 
> TO make things simple and to ensure things work, please first install this
> rpm using below steps:
> 
> 1) rpm -ivh '<src.rpm>' 
> 2) cd /root/rpmbuild/SOURCES/
> 3) tar -xvf dpdk-16.11.tar.xz
> 4) cd dpdk-16.11/
> 5) export DPDK_DIR=`pwd`
> 6) export DPDK_TARGET=x86_64-native-linuxapp-gcc
> 7) export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
> 8) make install T=$DPDK_TARGET DESTDIR=install
> 
> These 8 steps should get the DPDK libraries/apps built including bnxt PMD. 
> 
> Post these, please run the steps to configure OVS as per comment# 48.
> 
> Thanks
> Som

[root@en104 dpdk-16.11]# uname -a
Linux en104.dpdk.nfv 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux


[root@en104 SPECS]# cd /root/rpmbuild/SOURCES/
[root@en104 SOURCES]# l
total 8844
-rwxr-xr-x 1 root root     336 Feb 24  2017 dpdk-snapshot.sh*
-rw-rw-r-- 1 root root 9049848 Dec  8 02:58 dpdk-16.11.tar.xz
[root@en104 SOURCES]# tar -xvf dpdk-16.11.tar.xz
dpdk-16.11/
dpdk-16.11/.gitattributes
dpdk-16.11/.gitignore
...
...


[root@en104 SOURCES]# cd dpdk-16.11/
[root@en104 dpdk-16.11]# export DPDK_DIR=`pwd`
[root@en104 dpdk-16.11]# export DPDK_TARGET=x86_64-native-linuxapp-gcc
[root@en104 dpdk-16.11]# export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
[root@en104 dpdk-16.11]# make install T=$DPDK_TARGET DESTDIR=install
Configuration done
== Build lib
== Build lib/librte_compat
  SYMLINK-FILE include/rte_compat.h
== Build lib/librte_eal
== Build lib/librte_eal/common
  SYMLINK-FILE include/generic/rte_atomic.h
  SYMLINK-FILE include/generic/rte_byteorder.h
  SYMLINK-FILE include/generic/rte_cycles.h
  SYMLINK-FILE include/generic/rte_prefetch.h
  SYMLINK-FILE include/generic/rte_spinlock.h
...
...
  CC [M]  /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2288:2: error: too few arguments to function ‘ndo_dflt_bridge_getlink’
  return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
  ^
In file included from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/dst.h:13:0,
                 from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/sock.h:72,
                 from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/tcp.h:23,
                 from /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:34:
/usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/rtnetlink.h:115:12: note: declared here
 extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
            ^
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: At top level:
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2317:2: error: unknown field ‘ndo_set_vf_vlan’ specified in initializer
  .ndo_set_vf_vlan = igb_ndo_set_vf_vlan,
  ^
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2296:1: error: control reaches end of non-void function [-Werror=return-type]
 }
 ^
cc1: all warnings being treated as errors
make[10]: *** [/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o] Error 1
make[9]: *** [_module_/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni] Error 2
make[8]: *** [sub-make] Error 2
make[7]: *** [rte_kni.ko] Error 2
make[6]: *** [kni] Error 2
make[5]: *** [linuxapp] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2
[root@en104 dpdk-16.11]#

Comment 52 Somnath Kotur 2017-12-11 03:09:19 UTC
(In reply to Dilip Daya from comment #51)
> (In reply to Somnath Kotur from comment #50)
> > Dilip,
> >  Just in case, i have even provided the src rpm (with the patch
> > http://www.dpdk.org/browse/dpdk/commit/
> > ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 for bnxt PMD applied) 
> > 
> > TO make things simple and to ensure things work, please first install this
> > rpm using below steps:
> > 
> > 1) rpm -ivh '<src.rpm>' 
> > 2) cd /root/rpmbuild/SOURCES/
> > 3) tar -xvf dpdk-16.11.tar.xz
> > 4) cd dpdk-16.11/
> > 5) export DPDK_DIR=`pwd`
> > 6) export DPDK_TARGET=x86_64-native-linuxapp-gcc
> > 7) export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
> > 8) make install T=$DPDK_TARGET DESTDIR=install
> > 
> > These 8 steps should get the DPDK libraries/apps built including bnxt PMD. 
> > 
> > Post these, please run the steps to configure OVS as per comment# 48.
> > 
> > Thanks
> > Som
> 
> [root@en104 dpdk-16.11]# uname -a
> Linux en104.dpdk.nfv 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05
> EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> [root@en104 SPECS]# cd /root/rpmbuild/SOURCES/
> [root@en104 SOURCES]# l
> total 8844
> -rwxr-xr-x 1 root root     336 Feb 24  2017 dpdk-snapshot.sh*
> -rw-rw-r-- 1 root root 9049848 Dec  8 02:58 dpdk-16.11.tar.xz
> [root@en104 SOURCES]# tar -xvf dpdk-16.11.tar.xz
> dpdk-16.11/
> dpdk-16.11/.gitattributes
> dpdk-16.11/.gitignore
> ...
> ...
> 
> 
> [root@en104 SOURCES]# cd dpdk-16.11/
> [root@en104 dpdk-16.11]# export DPDK_DIR=`pwd`
> [root@en104 dpdk-16.11]# export DPDK_TARGET=x86_64-native-linuxapp-gcc
> [root@en104 dpdk-16.11]# export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
> [root@en104 dpdk-16.11]# make install T=$DPDK_TARGET DESTDIR=install
> Configuration done
> == Build lib
> == Build lib/librte_compat
>   SYMLINK-FILE include/rte_compat.h
> == Build lib/librte_eal
> == Build lib/librte_eal/common
>   SYMLINK-FILE include/generic/rte_atomic.h
>   SYMLINK-FILE include/generic/rte_byteorder.h
>   SYMLINK-FILE include/generic/rte_cycles.h
>   SYMLINK-FILE include/generic/rte_prefetch.h
>   SYMLINK-FILE include/generic/rte_spinlock.h
> ...
> ...
>   CC [M] 
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.o
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c:2288:2: error: too few arguments to
> function ‘ndo_dflt_bridge_getlink’
>   return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
>   ^
> In file included from
> /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/dst.h:13:0,
>                  from
> /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/sock.h:72,
>                  from
> /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/tcp.h:23,
>                  from
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c:34:
> /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/rtnetlink.h:115:12:
> note: declared here
>  extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
>             ^
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c: At top level:
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c:2317:2: error: unknown field
> ‘ndo_set_vf_vlan’ specified in initializer
>   .ndo_set_vf_vlan = igb_ndo_set_vf_vlan,
>   ^
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
> /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.c:2296:1: error: control reaches end of
> non-void function [-Werror=return-type]
>  }
>  ^
> cc1: all warnings being treated as errors
> make[10]: ***
> [/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/
> librte_eal/linuxapp/kni/igb_main.o] Error 1
> make[9]: ***
> [_module_/root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/
> lib/librte_eal/linuxapp/kni] Error 2
> make[8]: *** [sub-make] Error 2
> make[7]: *** [rte_kni.ko] Error 2
> make[6]: *** [kni] Error 2
> make[5]: *** [linuxapp] Error 2
> make[4]: *** [librte_eal] Error 2
> make[3]: *** [lib] Error 2
> make[2]: *** [all] Error 2
> make[1]: *** [pre_install] Error 2
> make: *** [install] Error 2
> [root@en104 dpdk-16.11]#

Hi Dilip,
          What distro are you working on ? Seems like that distro kernel version is incompatible with the igb driver especially the ".ndo_bridge_* " routines.
You could workaround this problem by setting the corresponding config variables to 'n' (open x86_64-native-linuxapp-gcc/.config  and set these 3 config variables to 'n')  and rebuild.
CONFIG_RTE_LIBRTE_KNI=n
CONFIG_RTE_LIBRTE_PMD_KNI=n
CONFIG_RTE_KNI_KMOD=n

Thanks
Som

Comment 53 Sandeep Raman 2017-12-11 09:29:50 UTC
Hi Som, Dilip,

Changed the following config variables to 'n'
CONFIG_RTE_LIBRTE_KNI=n
CONFIG_RTE_KNI_KMOD=n

CONFIG_RTE_LIBRTE_PMD_KNI was not present, did not add it.

The rebuild completed successfully.

[root@undercloud dpdk-16.11]# make install T=$DPDK_TARGET DESTDIR=install
Configuration done
Using local configuration
== Build lib
== Build lib/librte_compat
  SYMLINK-FILE include/rte_compat.h
== Build lib/librte_eal
== Build lib/librte_eal/common
  SYMLINK-FILE include/generic/rte_atomic.h
  SYMLINK-FILE include/generic/rte_byteorder.h
  SYMLINK-FILE include/generic/rte_cycles.h
  SYMLINK-FILE include/generic/rte_prefetch.h
  SYMLINK-FILE include/generic/rte_spinlock.h
  SYMLINK-FILE include/generic/rte_memcpy.h
  SYMLINK-FILE include/generic/rte_cpuflags.h
  SYMLINK-FILE include/generic/rte_rwlock.h
  SYMLINK-FILE include/rte_branch_prediction.h
  SYMLINK-FILE include/rte_common.h
  SYMLINK-FILE include/rte_debug.h
  SYMLINK-FILE include/rte_eal.h
  SYMLINK-FILE include/rte_errno.h
  SYMLINK-FILE include/rte_launch.h
  SYMLINK-FILE include/rte_lcore.h
  SYMLINK-FILE include/rte_log.h
  SYMLINK-FILE include/rte_memory.h
  SYMLINK-FILE include/rte_memzone.h
  SYMLINK-FILE include/rte_pci.h
  SYMLINK-FILE include/rte_per_lcore.h
  SYMLINK-FILE include/rte_random.h
  SYMLINK-FILE include/rte_tailq.h
  SYMLINK-FILE include/rte_interrupts.h
  SYMLINK-FILE include/rte_alarm.h
  SYMLINK-FILE include/rte_string_fns.h
  SYMLINK-FILE include/rte_version.h
  SYMLINK-FILE include/rte_eal_memconfig.h
  SYMLINK-FILE include/rte_malloc_heap.h
  SYMLINK-FILE include/rte_hexdump.h
  SYMLINK-FILE include/rte_devargs.h
  SYMLINK-FILE include/rte_dev.h
  SYMLINK-FILE include/rte_vdev.h
  SYMLINK-FILE include/rte_pci_dev_feature_defs.h
  SYMLINK-FILE include/rte_pci_dev_features.h
  SYMLINK-FILE include/rte_malloc.h
  SYMLINK-FILE include/rte_keepalive.h
  SYMLINK-FILE include/rte_time.h
  SYMLINK-FILE include/rte_rwlock.h
  SYMLINK-FILE include/rte_memcpy.h
  SYMLINK-FILE include/rte_cycles.h
  SYMLINK-FILE include/rte_spinlock.h
  SYMLINK-FILE include/rte_atomic_32.h
  SYMLINK-FILE include/rte_vect.h
  SYMLINK-FILE include/rte_prefetch.h
  SYMLINK-FILE include/rte_byteorder_32.h
  SYMLINK-FILE include/rte_atomic_64.h
  SYMLINK-FILE include/rte_rtm.h
  SYMLINK-FILE include/rte_cpuflags.h
  SYMLINK-FILE include/rte_byteorder_64.h
  SYMLINK-FILE include/rte_atomic.h
  SYMLINK-FILE include/rte_byteorder.h
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/eal
  CC eal.o
  CC eal_hugepage_info.o
  CC eal_memory.o
  CC eal_thread.o
  CC eal_log.o
  CC eal_vfio.o
  CC eal_vfio_mp_sync.o
  CC eal_pci.o
  CC eal_pci_uio.o
  CC eal_pci_vfio.o
  CC eal_debug.o
  CC eal_lcore.o
  CC eal_timer.o
  CC eal_interrupts.o
  CC eal_alarm.o
  CC eal_common_lcore.o
  CC eal_common_timer.o
  CC eal_common_memzone.o
  CC eal_common_log.o
  CC eal_common_launch.o
  CC eal_common_vdev.o
  CC eal_common_pci.o
  CC eal_common_pci_uio.o
  CC eal_common_memory.o
  CC eal_common_tailqs.o
  CC eal_common_errno.o
  CC eal_common_cpuflags.o
  CC eal_common_string_fns.o
  CC eal_common_hexdump.o
  CC eal_common_devargs.o
  CC eal_common_dev.o
  CC eal_common_options.o
  CC eal_common_thread.o
  CC eal_common_proc.o
  CC rte_malloc.o
  CC malloc_elem.o
  CC malloc_heap.o
  CC rte_keepalive.o
  CC rte_cpuflags.o
  CC rte_spinlock.o
  AR librte_eal.a
  SYMLINK-FILE include/exec-env/rte_interrupts.h
  SYMLINK-FILE include/exec-env/rte_kni_common.h
  SYMLINK-FILE include/exec-env/rte_dom0_common.h
  INSTALL-LIB librte_eal.a
== Build lib/librte_eal/linuxapp/igb_uio
  LD      /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/built-in.o
  CC [M]  /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.mod.o
  LD [M]  /root/rpmbuild/SOURCES/dpdk-16.11/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko
INSTALL-MODULE igb_uio.ko
== Build lib/librte_ring
  CC rte_ring.o
  AR librte_ring.a
  SYMLINK-FILE include/rte_ring.h
  INSTALL-LIB librte_ring.a
== Build lib/librte_mempool
  CC rte_mempool.o
  CC rte_mempool_ops.o
  CC rte_mempool_ring.o
  CC rte_mempool_stack.o
  AR librte_mempool.a
  SYMLINK-FILE include/rte_mempool.h
  INSTALL-LIB librte_mempool.a
== Build lib/librte_mbuf
  CC rte_mbuf.o
  CC rte_mbuf_ptype.o
  AR librte_mbuf.a
  SYMLINK-FILE include/rte_mbuf.h
  SYMLINK-FILE include/rte_mbuf_ptype.h
  INSTALL-LIB librte_mbuf.a
== Build lib/librte_timer
  CC rte_timer.o
  AR librte_timer.a
  SYMLINK-FILE include/rte_timer.h
  INSTALL-LIB librte_timer.a
== Build lib/librte_cfgfile
  CC rte_cfgfile.o
  AR librte_cfgfile.a
  SYMLINK-FILE include/rte_cfgfile.h
  INSTALL-LIB librte_cfgfile.a
== Build lib/librte_cmdline
  CC cmdline.o
  CC cmdline_cirbuf.o
  CC cmdline_parse.o
  CC cmdline_parse_etheraddr.o
  CC cmdline_parse_ipaddr.o
  CC cmdline_parse_num.o
  CC cmdline_parse_string.o
  CC cmdline_rdline.o
  CC cmdline_vt100.o
  CC cmdline_socket.o
  CC cmdline_parse_portlist.o
  AR librte_cmdline.a
  SYMLINK-FILE include/cmdline.h
  SYMLINK-FILE include/cmdline_parse.h
  SYMLINK-FILE include/cmdline_parse_num.h
  SYMLINK-FILE include/cmdline_parse_ipaddr.h
  SYMLINK-FILE include/cmdline_parse_etheraddr.h
  SYMLINK-FILE include/cmdline_parse_string.h
  SYMLINK-FILE include/cmdline_rdline.h
  SYMLINK-FILE include/cmdline_vt100.h
  SYMLINK-FILE include/cmdline_socket.h
  SYMLINK-FILE include/cmdline_cirbuf.h
  SYMLINK-FILE include/cmdline_parse_portlist.h
  INSTALL-LIB librte_cmdline.a
== Build lib/librte_net
  CC rte_net.o
  AR librte_net.a
  SYMLINK-FILE include/rte_ip.h
  SYMLINK-FILE include/rte_tcp.h
  SYMLINK-FILE include/rte_udp.h
  SYMLINK-FILE include/rte_sctp.h
  SYMLINK-FILE include/rte_icmp.h
  SYMLINK-FILE include/rte_arp.h
  SYMLINK-FILE include/rte_ether.h
  SYMLINK-FILE include/rte_gre.h
  SYMLINK-FILE include/rte_net.h
  INSTALL-LIB librte_net.a
== Build lib/librte_ether
  CC rte_ethdev.o
  AR librte_ethdev.a
  SYMLINK-FILE include/rte_ethdev.h
  SYMLINK-FILE include/rte_eth_ctrl.h
  SYMLINK-FILE include/rte_dev_info.h
  INSTALL-LIB librte_ethdev.a
== Build lib/librte_kvargs
  CC rte_kvargs.o
  AR librte_kvargs.a
  SYMLINK-FILE include/rte_kvargs.h
  INSTALL-LIB librte_kvargs.a
== Build lib/librte_cryptodev
  CC rte_cryptodev.o
  AR librte_cryptodev.a
  SYMLINK-FILE include/rte_crypto.h
  SYMLINK-FILE include/rte_crypto_sym.h
  SYMLINK-FILE include/rte_cryptodev.h
  SYMLINK-FILE include/rte_cryptodev_pmd.h
  INSTALL-LIB librte_cryptodev.a
== Build lib/librte_vhost
  CC fd_man.o
  CC socket.o
  CC vhost.o
  CC vhost_user.o
  CC virtio_net.o
  AR librte_vhost.a
  SYMLINK-FILE include/rte_virtio_net.h
  INSTALL-LIB librte_vhost.a
== Build lib/librte_hash
  CC rte_cuckoo_hash.o
  CC rte_fbk_hash.o
  AR librte_hash.a
  SYMLINK-FILE include/rte_hash.h
  SYMLINK-FILE include/rte_hash_crc.h
  SYMLINK-FILE include/rte_jhash.h
  SYMLINK-FILE include/rte_thash.h
  SYMLINK-FILE include/rte_fbk_hash.h
  INSTALL-LIB librte_hash.a
== Build lib/librte_lpm
  CC rte_lpm.o
  CC rte_lpm6.o
  AR librte_lpm.a
  SYMLINK-FILE include/rte_lpm.h
  SYMLINK-FILE include/rte_lpm6.h
  SYMLINK-FILE include/rte_lpm_sse.h
  INSTALL-LIB librte_lpm.a
== Build lib/librte_acl
  CC tb_mem.o
  CC rte_acl.o
  CC acl_bld.o
  CC acl_gen.o
  CC acl_run_scalar.o
  CC acl_run_sse.o
  CC acl_run_avx2.o
  AR librte_acl.a
  SYMLINK-FILE include/rte_acl_osdep.h
  SYMLINK-FILE include/rte_acl.h
  INSTALL-LIB librte_acl.a
== Build lib/librte_ip_frag
  CC rte_ipv4_fragmentation.o
  CC rte_ipv6_fragmentation.o
  CC rte_ipv4_reassembly.o
  CC rte_ipv6_reassembly.o
  CC rte_ip_frag_common.o
  CC ip_frag_internal.o
  AR librte_ip_frag.a
  SYMLINK-FILE include/rte_ip_frag.h
  INSTALL-LIB librte_ip_frag.a
== Build lib/librte_jobstats
  CC rte_jobstats.o
  AR librte_jobstats.a
  SYMLINK-FILE include/rte_jobstats.h
  INSTALL-LIB librte_jobstats.a
== Build lib/librte_power
  CC rte_power.o
  CC rte_power_acpi_cpufreq.o
  CC rte_power_kvm_vm.o
  CC guest_channel.o
  AR librte_power.a
  SYMLINK-FILE include/rte_power.h
  INSTALL-LIB librte_power.a
== Build lib/librte_meter
  CC rte_meter.o
  AR librte_meter.a
  SYMLINK-FILE include/rte_meter.h
  INSTALL-LIB librte_meter.a
== Build lib/librte_sched
  CC rte_sched.o
  CC rte_red.o
  CC rte_approx.o
  CC rte_reciprocal.o
  AR librte_sched.a
  SYMLINK-FILE include/rte_sched.h
  SYMLINK-FILE include/rte_bitmap.h
  SYMLINK-FILE include/rte_sched_common.h
  SYMLINK-FILE include/rte_red.h
  SYMLINK-FILE include/rte_approx.h
  SYMLINK-FILE include/rte_reciprocal.h
  INSTALL-LIB librte_sched.a
== Build lib/librte_distributor
  CC rte_distributor.o
  AR librte_distributor.a
  SYMLINK-FILE include/rte_distributor.h
  INSTALL-LIB librte_distributor.a
== Build lib/librte_port
  CC rte_port_ethdev.o
  CC rte_port_ring.o
  CC rte_port_frag.o
  CC rte_port_ras.o
  CC rte_port_sched.o
  CC rte_port_fd.o
  CC rte_port_source_sink.o
  AR librte_port.a
  SYMLINK-FILE include/rte_port.h
  SYMLINK-FILE include/rte_port_ethdev.h
  SYMLINK-FILE include/rte_port_ring.h
  SYMLINK-FILE include/rte_port_frag.h
  SYMLINK-FILE include/rte_port_ras.h
  SYMLINK-FILE include/rte_port_sched.h
  SYMLINK-FILE include/rte_port_fd.h
  SYMLINK-FILE include/rte_port_source_sink.h
  INSTALL-LIB librte_port.a
== Build lib/librte_table
  CC rte_table_lpm.o
  CC rte_table_lpm_ipv6.o
  CC rte_table_acl.o
  CC rte_table_hash_cuckoo.o
  CC rte_table_hash_key8.o
  CC rte_table_hash_key16.o
  CC rte_table_hash_key32.o
  CC rte_table_hash_ext.o
  CC rte_table_hash_lru.o
  CC rte_table_array.o
  CC rte_table_stub.o
  AR librte_table.a
  SYMLINK-FILE include/rte_table.h
  SYMLINK-FILE include/rte_table_lpm.h
  SYMLINK-FILE include/rte_table_lpm_ipv6.h
  SYMLINK-FILE include/rte_table_acl.h
  SYMLINK-FILE include/rte_table_hash.h
  SYMLINK-FILE include/rte_lru.h
  SYMLINK-FILE include/rte_table_array.h
  SYMLINK-FILE include/rte_table_stub.h
  INSTALL-LIB librte_table.a
== Build lib/librte_pipeline
  CC rte_pipeline.o
  AR librte_pipeline.a
  SYMLINK-FILE include/rte_pipeline.h
  INSTALL-LIB librte_pipeline.a
== Build lib/librte_reorder
  CC rte_reorder.o
  AR librte_reorder.a
  SYMLINK-FILE include/rte_reorder.h
  INSTALL-LIB librte_reorder.a
== Build lib/librte_pdump
  CC rte_pdump.o
  AR librte_pdump.a
  SYMLINK-FILE include/rte_pdump.h
  INSTALL-LIB librte_pdump.a
== Build buildtools
== Build buildtools/pmdinfogen
  HOSTCC pmdinfogen.o
  HOSTLD dpdk-pmdinfogen
  INSTALL-HOSTAPP dpdk-pmdinfogen
== Build drivers
== Build drivers/net
== Build drivers/net/af_packet
  CC rte_eth_af_packet.o
  PMDINFO rte_eth_af_packet.o.pmd.c
  CC rte_eth_af_packet.o.pmd.o
  LD rte_eth_af_packet.o
  AR librte_pmd_af_packet.a
  INSTALL-LIB librte_pmd_af_packet.a
== Build drivers/net/bonding
  CC rte_eth_bond_api.o
  CC rte_eth_bond_pmd.o
  PMDINFO rte_eth_bond_pmd.o.pmd.c
  CC rte_eth_bond_pmd.o.pmd.o
  LD rte_eth_bond_pmd.o
  CC rte_eth_bond_args.o
  CC rte_eth_bond_8023ad.o
  CC rte_eth_bond_alb.o
  AR librte_pmd_bond.a
  SYMLINK-FILE include/rte_eth_bond.h
  SYMLINK-FILE include/rte_eth_bond_8023ad.h
  INSTALL-LIB librte_pmd_bond.a
== Build drivers/net/cxgbe
  CC cxgbe_ethdev.o
  PMDINFO cxgbe_ethdev.o.pmd.c
  CC cxgbe_ethdev.o.pmd.o
  LD cxgbe_ethdev.o
  CC cxgbe_main.o
  CC sge.o
  CC t4_hw.o
  AR librte_pmd_cxgbe.a
  INSTALL-LIB librte_pmd_cxgbe.a
== Build drivers/net/e1000
  CC e1000_80003es2lan.o
  CC e1000_82540.o
  CC e1000_82541.o
  CC e1000_82542.o
  CC e1000_82543.o
  CC e1000_82571.o
  CC e1000_82575.o
  CC e1000_i210.o
  CC e1000_api.o
  CC e1000_ich8lan.o
  CC e1000_mac.o
  CC e1000_manage.o
  CC e1000_mbx.o
  CC e1000_nvm.o
  CC e1000_osdep.o
  CC e1000_phy.o
  CC e1000_vf.o
  CC igb_ethdev.o
  PMDINFO igb_ethdev.o.pmd.c
  CC igb_ethdev.o.pmd.o
  LD igb_ethdev.o
  CC igb_rxtx.o
  CC igb_pf.o
  CC em_ethdev.o
  PMDINFO em_ethdev.o.pmd.c
  CC em_ethdev.o.pmd.o
  LD em_ethdev.o
  CC em_rxtx.o
  AR librte_pmd_e1000.a
  INSTALL-LIB librte_pmd_e1000.a
== Build drivers/net/ena
  CC ena_ethdev.o
  PMDINFO ena_ethdev.o.pmd.c
  CC ena_ethdev.o.pmd.o
  LD ena_ethdev.o
  CC ena_com.o
  CC ena_eth_com.o
  AR librte_pmd_ena.a
  INSTALL-LIB librte_pmd_ena.a
== Build drivers/net/enic
  CC enic_ethdev.o
  PMDINFO enic_ethdev.o.pmd.c
  CC enic_ethdev.o.pmd.o
  LD enic_ethdev.o
  CC enic_main.o
  CC enic_rxtx.o
  CC enic_clsf.o
  CC enic_res.o
  CC base/vnic_cq.o
  CC base/vnic_wq.o
  CC base/vnic_dev.o
  CC base/vnic_intr.o
  CC base/vnic_rq.o
  CC base/vnic_rss.o
  AR librte_pmd_enic.a
  INSTALL-LIB librte_pmd_enic.a
== Build drivers/net/fm10k
  CC fm10k_ethdev.o
  PMDINFO fm10k_ethdev.o.pmd.c
  CC fm10k_ethdev.o.pmd.o
  LD fm10k_ethdev.o
  CC fm10k_rxtx.o
  CC fm10k_pf.o
  CC fm10k_tlv.o
  CC fm10k_common.o
  CC fm10k_mbx.o
  CC fm10k_vf.o
  CC fm10k_api.o
  CC fm10k_rxtx_vec.o
  AR librte_pmd_fm10k.a
  INSTALL-LIB librte_pmd_fm10k.a
== Build drivers/net/i40e
  CC i40e_adminq.o
  CC i40e_common.o
  CC i40e_diag.o
  CC i40e_hmc.o
  CC i40e_lan_hmc.o
  CC i40e_nvm.o
  CC i40e_dcb.o
  CC i40e_ethdev.o
  PMDINFO i40e_ethdev.o.pmd.c
  CC i40e_ethdev.o.pmd.o
  LD i40e_ethdev.o
  CC i40e_rxtx.o
  CC i40e_rxtx_vec_sse.o
  CC i40e_ethdev_vf.o
  PMDINFO i40e_ethdev_vf.o.pmd.c
  CC i40e_ethdev_vf.o.pmd.o
  LD i40e_ethdev_vf.o
  CC i40e_pf.o
  CC i40e_fdir.o
  AR librte_pmd_i40e.a
  INSTALL-LIB librte_pmd_i40e.a
== Build drivers/net/ixgbe
  CC ixgbe_common.o
  CC ixgbe_82598.o
  CC ixgbe_82599.o
  CC ixgbe_x540.o
  CC ixgbe_x550.o
  CC ixgbe_phy.o
  CC ixgbe_api.o
  CC ixgbe_vf.o
  CC ixgbe_dcb.o
  CC ixgbe_dcb_82599.o
  CC ixgbe_dcb_82598.o
  CC ixgbe_mbx.o
  CC ixgbe_rxtx.o
  CC ixgbe_ethdev.o
  PMDINFO ixgbe_ethdev.o.pmd.c
  CC ixgbe_ethdev.o.pmd.o
  LD ixgbe_ethdev.o
  CC ixgbe_fdir.o
  CC ixgbe_pf.o
  CC ixgbe_rxtx_vec_sse.o
  AR librte_pmd_ixgbe.a
  SYMLINK-FILE include/rte_pmd_ixgbe.h
  INSTALL-LIB librte_pmd_ixgbe.a
== Build drivers/net/bnxt
  CC bnxt_cpr.o
  CC bnxt_ethdev.o
  PMDINFO bnxt_ethdev.o.pmd.c
  CC bnxt_ethdev.o.pmd.o
  LD bnxt_ethdev.o
  CC bnxt_filter.o
  CC bnxt_hwrm.o
  CC bnxt_ring.o
  CC bnxt_rxq.o
  CC bnxt_rxr.o
  CC bnxt_stats.o
  CC bnxt_txq.o
  CC bnxt_txr.o
  CC bnxt_vnic.o
  CC bnxt_irq.o
  AR librte_pmd_bnxt.a
  INSTALL-LIB librte_pmd_bnxt.a
== Build drivers/net/null
  CC rte_eth_null.o
  PMDINFO rte_eth_null.o.pmd.c
  CC rte_eth_null.o.pmd.o
  LD rte_eth_null.o
  AR librte_pmd_null.a
  SYMLINK-FILE include/rte_eth_null.h
  INSTALL-LIB librte_pmd_null.a
== Build drivers/net/qede
  CC base/ecore_dev.o
  CC base/ecore_hw.o
  CC base/ecore_cxt.o
  CC base/ecore_l2.o
  CC base/ecore_sp_commands.o
  CC base/ecore_init_fw_funcs.o
  CC base/ecore_spq.o
  CC base/ecore_init_ops.o
  CC base/ecore_mcp.o
  CC base/ecore_int.o
  CC base/ecore_dcbx.o
  CC base/bcm_osal.o
  CC base/ecore_sriov.o
  CC base/ecore_vf.o
  CC qede_ethdev.o
  PMDINFO qede_ethdev.o.pmd.c
  CC qede_ethdev.o.pmd.o
  LD qede_ethdev.o
  CC qede_eth_if.o
  CC qede_main.o
  CC qede_rxtx.o
  AR librte_pmd_qede.a
  INSTALL-LIB librte_pmd_qede.a
== Build drivers/net/ring
  CC rte_eth_ring.o
  PMDINFO rte_eth_ring.o.pmd.c
  CC rte_eth_ring.o.pmd.o
  LD rte_eth_ring.o
  AR librte_pmd_ring.a
  SYMLINK-FILE include/rte_eth_ring.h
  INSTALL-LIB librte_pmd_ring.a
== Build drivers/net/virtio
  CC virtqueue.o
  CC virtio_pci.o
  CC virtio_rxtx.o
  CC virtio_ethdev.o
  PMDINFO virtio_ethdev.o.pmd.c
  CC virtio_ethdev.o.pmd.o
  LD virtio_ethdev.o
  CC virtio_rxtx_simple.o
  CC virtio_rxtx_simple_sse.o
  CC virtio_user/vhost_user.o
  CC virtio_user/virtio_user_dev.o
  CC virtio_user_ethdev.o
  PMDINFO virtio_user_ethdev.o.pmd.c
  CC virtio_user_ethdev.o.pmd.o
  LD virtio_user_ethdev.o
  AR librte_pmd_virtio.a
  INSTALL-LIB librte_pmd_virtio.a
== Build drivers/net/vmxnet3
  CC vmxnet3_rxtx.o
  CC vmxnet3_ethdev.o
  PMDINFO vmxnet3_ethdev.o.pmd.c
  CC vmxnet3_ethdev.o.pmd.o
  LD vmxnet3_ethdev.o
  AR librte_pmd_vmxnet3_uio.a
  INSTALL-LIB librte_pmd_vmxnet3_uio.a
== Build drivers/net/vhost
  CC rte_eth_vhost.o
  PMDINFO rte_eth_vhost.o.pmd.c
  CC rte_eth_vhost.o.pmd.o
  LD rte_eth_vhost.o
  AR librte_pmd_vhost.a
  SYMLINK-FILE include/rte_eth_vhost.h
  INSTALL-LIB librte_pmd_vhost.a
== Build drivers/crypto
== Build drivers/crypto/null
  CC null_crypto_pmd.o
  PMDINFO null_crypto_pmd.o.pmd.c
  CC null_crypto_pmd.o.pmd.o
  LD null_crypto_pmd.o
  CC null_crypto_pmd_ops.o
  AR librte_pmd_null_crypto.a
  INSTALL-LIB librte_pmd_null_crypto.a
== Build app
== Build app/test
  CC commands.o
  CC test.o
  CC resource.o
  CC test_resource.o
  MKRES test_resource_c.res.o
  CC test_prefetch.o
  CC test_byteorder.o
  CC test_per_lcore.o
  CC test_atomic.o
  CC test_malloc.o
  CC test_cycles.o
  CC test_spinlock.o
  CC test_memory.o
  CC test_memzone.o
  CC test_ring.o
  CC test_ring_perf.o
  CC test_pmd_perf.o
  CC test_table.o
  CC test_table_pipeline.o
  CC test_table_tables.o
  CC test_table_ports.o
  CC test_table_combined.o
  CC test_table_acl.o
  CC test_rwlock.o
  CC test_timer.o
  CC test_timer_perf.o
  CC test_timer_racecond.o
  CC test_mempool.o
  CC test_mempool_perf.o
  CC test_mbuf.o
  CC test_logs.o
  CC test_memcpy.o
  CC test_memcpy_perf.o
  CC test_hash.o
  CC test_thash.o
  CC test_hash_perf.o
  CC test_hash_functions.o
  CC test_hash_scaling.o
  CC test_hash_multiwriter.o
  CC test_lpm.o
  CC test_lpm_perf.o
  CC test_lpm6.o
  CC test_lpm6_perf.o
  CC test_debug.o
  CC test_errno.o
  CC test_tailq.o
  CC test_string_fns.o
  CC test_cpuflags.o
  CC test_mp_secondary.o
  CC test_eal_flags.o
  CC test_eal_fs.o
  CC test_alarm.o
  CC test_interrupts.o
  CC test_version.o
  CC test_func_reentrancy.o
  CC test_cmdline.o
  CC test_cmdline_num.o
  CC test_cmdline_etheraddr.o
  CC test_cmdline_portlist.o
  CC test_cmdline_ipaddr.o
  CC test_cmdline_cirbuf.o
  CC test_cmdline_string.o
  CC test_cmdline_lib.o
  CC test_red.o
  CC test_sched.o
  CC test_meter.o
  CC test_power.o
  CC test_power_acpi_cpufreq.o
  CC test_power_kvm_vm.o
  CC test_common.o
  CC test_distributor.o
  CC test_distributor_perf.o
  CC test_reorder.o
  CC test_devargs.o
  CC virtual_pmd.o
  CC packet_burst_generator.o
  CC test_acl.o
  CC test_link_bonding.o
  CC test_link_bonding_mode4.o
  CC test_link_bonding_rssconf.o
  CC test_pmd_ring.o
  CC test_pmd_ring_perf.o
  CC test_cryptodev_blockcipher.o
  CC test_cryptodev_perf.o
  CC test_cryptodev.o
  CC test_kvargs.o
  LD test
  INSTALL-APP test
  INSTALL-MAP test.map
== Build app/test-acl
  CC main.o
  LD testacl
  INSTALL-APP testacl
  INSTALL-MAP testacl.map
== Build app/test-pipeline
  CC main.o
  CC config.o
  CC init.o
  CC runtime.o
  CC pipeline_stub.o
  CC pipeline_hash.o
  CC pipeline_lpm.o
  CC pipeline_lpm_ipv6.o
  CC pipeline_acl.o
  LD testpipeline
  INSTALL-APP testpipeline
  INSTALL-MAP testpipeline.map
== Build app/test-pmd
  CC testpmd.o
  CC parameters.o
  CC cmdline.o
  CC config.o
  CC iofwd.o
  CC macfwd.o
  CC macswap.o
  CC flowgen.o
  CC rxonly.o
  CC txonly.o
  CC csumonly.o
  CC icmpecho.o
  LD testpmd
  INSTALL-APP testpmd
  INSTALL-MAP testpmd.map
== Build app/cmdline_test
  CC cmdline_test.o
  CC commands.o
  LD cmdline_test
  INSTALL-APP cmdline_test
  INSTALL-MAP cmdline_test.map
== Build app/proc_info
  CC main.o
  LD dpdk-procinfo
  INSTALL-APP dpdk-procinfo
  INSTALL-MAP dpdk-procinfo.map
== Build app/pdump
  CC main.o
  LD dpdk-pdump
  INSTALL-APP dpdk-pdump
  INSTALL-MAP dpdk-pdump.map
Build complete [x86_64-native-linuxapp-gcc]
================== Installing install/
Installation in install/ complete
[root@undercloud dpdk-16.11]#


Regards,
Sandeep.

Comment 54 Dilip Daya 2017-12-11 20:08:09 UTC
(In reply to Sandeep Raman from comment #53)
> Hi Som, Dilip,
> 
> Changed the following config variables to 'n'
> CONFIG_RTE_LIBRTE_KNI=n
> CONFIG_RTE_KNI_KMOD=n
> 
> CONFIG_RTE_LIBRTE_PMD_KNI was not present, did not add it.
> 
> The rebuild completed successfully.

NOTE:  Environment is RHEL 7.4 (kernel 3.10.0-693.11.1.el7.x86_64)

[root@en104 config]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)

[root@en104 config]# uname -a
Linux en104.dpdk.nfv 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@en104 dilip]# rpm -Uvh dpdk-16.11-3.el7.src.rpm
Updating / installing...
   1:dpdk-16.11-3.el7                 ################################# [100%]
[root@en104 dilip]# cd ~/rpmbuild/SOURCES/
[root@en104 SOURCES]# l
total 8844
-rw-rw-r-- 1 root root 9048572 Dec  8  2016 dpdk-16.11.tar.xz
-rwxr-xr-x 1 root root     336 Feb 24  2017 dpdk-snapshot.sh*

[root@en104 SOURCES]# tar -xf dpdk-16.11.tar.xz 
[root@en104 SOURCES]# l
total 8844
drwxrwxr-x 13 root root     298 Nov 13  2016 dpdk-16.11/
-rw-rw-r--  1 root root 9048572 Dec  8  2016 dpdk-16.11.tar.xz
-rwxr-xr-x  1 root root     336 Feb 24  2017 dpdk-snapshot.sh*

[root@en104 SOURCES]# cd dpdk-16.11/
[root@en104 dpdk-16.11]# export DPDK_DIR=`pwd`
[root@en104 dpdk-16.11]# export DPDK_TARGET=x86_64-native-linuxapp-gcc
[root@en104 dpdk-16.11]# export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET

[root@en104 dpdk-16.11]# cd config/
[root@en104 config]# grep CONFIG_RTE_LIBRTE_KNI *
common_base:CONFIG_RTE_LIBRTE_KNI=n
common_linuxapp:CONFIG_RTE_LIBRTE_KNI=y
defconfig_arm-armv7a-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
defconfig_i686-native-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
defconfig_i686-native-linuxapp-icc:CONFIG_RTE_LIBRTE_KNI=n
defconfig_tile-tilegx-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
defconfig_x86_x32-native-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n

[root@en104 config]# grep CONFIG_RTE_KNI_KMOD *
common_base:CONFIG_RTE_KNI_KMOD=n
common_linuxapp:CONFIG_RTE_KNI_KMOD=y
defconfig_arm-armv7a-linuxapp-gcc:CONFIG_RTE_KNI_KMOD=n
defconfig_tile-tilegx-linuxapp-gcc:CONFIG_RTE_KNI_KMOD=n
[root@en104 config]# grep CONFIG_RTE_LIBRTE_PMD_KNI *
[root@en104 config]# 

Changed to:
common_linuxapp:CONFIG_RTE_LIBRTE_KNI=n
common_linuxapp:CONFIG_RTE_KNI_KMOD=n

[root@en104 config]# cd ..
[root@en104 dpdk-16.11]# make install T=$DPDK_TARGET DESTDIR=install
...
...
  INSTALL-MAP dpdk-pdump.map
Build complete [x86_64-native-linuxapp-gcc]
================== Installing install/
Installation in install/ complete
[root@en104 dpdk-16.11]# 


[root@en104 dpdk-16.11]# tools/dpdk-devbind.py --bind vfio-pci 0000:12:00.0 0000:12:00.1
[root@en104 dpdk-16.11]# tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


[root@en104 dpdk-16.11]# ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
Interactive-mode selected
EAL: Error - exiting with code: 1
  Cause: Invalid port 2
[root@en104 dpdk-16.11]#

Comment 55 Dilip Daya 2017-12-11 20:29:12 UTC
(In reply to Dilip Daya from comment #54)
> (In reply to Sandeep Raman from comment #53)
> > Hi Som, Dilip,
> > 
> > Changed the following config variables to 'n'
> > CONFIG_RTE_LIBRTE_KNI=n
> > CONFIG_RTE_KNI_KMOD=n
> > 
> > CONFIG_RTE_LIBRTE_PMD_KNI was not present, did not add it.
> > 
> > The rebuild completed successfully.
> 
> NOTE:  Environment is RHEL 7.4 (kernel 3.10.0-693.11.1.el7.x86_64)
> 
> [root@en104 config]# cat /etc/redhat-release 
> Red Hat Enterprise Linux Server release 7.4 (Maipo)
> 
> [root@en104 config]# uname -a
> Linux en104.dpdk.nfv 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05
> EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> 
> [root@en104 dilip]# rpm -Uvh dpdk-16.11-3.el7.src.rpm
> Updating / installing...
>    1:dpdk-16.11-3.el7                 #################################
> [100%]
> [root@en104 dilip]# cd ~/rpmbuild/SOURCES/
> [root@en104 SOURCES]# l
> total 8844
> -rw-rw-r-- 1 root root 9048572 Dec  8  2016 dpdk-16.11.tar.xz
> -rwxr-xr-x 1 root root     336 Feb 24  2017 dpdk-snapshot.sh*
> 
> [root@en104 SOURCES]# tar -xf dpdk-16.11.tar.xz 
> [root@en104 SOURCES]# l
> total 8844
> drwxrwxr-x 13 root root     298 Nov 13  2016 dpdk-16.11/
> -rw-rw-r--  1 root root 9048572 Dec  8  2016 dpdk-16.11.tar.xz
> -rwxr-xr-x  1 root root     336 Feb 24  2017 dpdk-snapshot.sh*
> 
> [root@en104 SOURCES]# cd dpdk-16.11/
> [root@en104 dpdk-16.11]# export DPDK_DIR=`pwd`
> [root@en104 dpdk-16.11]# export DPDK_TARGET=x86_64-native-linuxapp-gcc
> [root@en104 dpdk-16.11]# export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
> 
> [root@en104 dpdk-16.11]# cd config/
> [root@en104 config]# grep CONFIG_RTE_LIBRTE_KNI *
> common_base:CONFIG_RTE_LIBRTE_KNI=n
> common_linuxapp:CONFIG_RTE_LIBRTE_KNI=y
> defconfig_arm-armv7a-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
> defconfig_i686-native-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
> defconfig_i686-native-linuxapp-icc:CONFIG_RTE_LIBRTE_KNI=n
> defconfig_tile-tilegx-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
> defconfig_x86_x32-native-linuxapp-gcc:CONFIG_RTE_LIBRTE_KNI=n
> 
> [root@en104 config]# grep CONFIG_RTE_KNI_KMOD *
> common_base:CONFIG_RTE_KNI_KMOD=n
> common_linuxapp:CONFIG_RTE_KNI_KMOD=y
> defconfig_arm-armv7a-linuxapp-gcc:CONFIG_RTE_KNI_KMOD=n
> defconfig_tile-tilegx-linuxapp-gcc:CONFIG_RTE_KNI_KMOD=n
> [root@en104 config]# grep CONFIG_RTE_LIBRTE_PMD_KNI *
> [root@en104 config]# 
> 
> Changed to:
> common_linuxapp:CONFIG_RTE_LIBRTE_KNI=n
> common_linuxapp:CONFIG_RTE_KNI_KMOD=n
> 
> [root@en104 config]# cd ..
> [root@en104 dpdk-16.11]# make install T=$DPDK_TARGET DESTDIR=install
> ...
> ...
>   INSTALL-MAP dpdk-pdump.map
> Build complete [x86_64-native-linuxapp-gcc]
> ================== Installing install/
> Installation in install/ complete
> [root@en104 dpdk-16.11]# 
> 
> 
> [root@en104 dpdk-16.11]# tools/dpdk-devbind.py --bind vfio-pci 0000:12:00.0
> 0000:12:00.1
> [root@en104 dpdk-16.11]# tools/dpdk-devbind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 
> 
> [root@en104 dpdk-16.11]# ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe -n
> 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2
> --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1
> --socket-num=0
> EAL: Detected 32 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: No probed ethernet devices
> Interactive-mode selected
> EAL: Error - exiting with code: 1
>   Cause: Invalid port 2
> [root@en104 dpdk-16.11]#

I then applied patch:
http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415

and successfully rebuilt the above.

[root@en104 dpdk-16.11]# tools/dpdk-devbind.py --bind vfio-pci 0000:12:00.0 0000:12:00.1
[root@en104 dpdk-16.11]# tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en

Network devices using kernel driver
===================================
0000:02:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=vfio-pci 
0000:02:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=vfio-pci 
0000:02:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=vfio-pci 
0000:02:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=vfio-pci 
0000:5d:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' if=eno5 drv=bnxt_en unused=vfio-pci 
0000:5d:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' if=eno6d1 drv=bnxt_en unused=vfio-pci 

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
[root@en104 dpdk-16.11]# 
[root@en104 dpdk-16.11]# ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe -n 4 -w 0000:12:00.0 -w 0000:12:00.1 -- -i --nb-cores=2 --nb-ports=2 --forward-mode=io --rxd=4096 --txd=4096 --burst=64 --rxq=1 --txq=1 --socket-num=0
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7fad53600000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7fad55dd1000M
Interactive-mode selected
Set io packet forwarding mode
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
Configuring Port 0 (socket 0)
PMD: Port 0 Link Up - speed 25000 Mbps - full-duplex
Port 0: 9C:DC:71:B3:B8:A0
Configuring Port 1 (socket 0)
PMD: Port 1 Link Up - speed 25000 Mbps - full-duplex
Port 1: 9C:DC:71:B3:B8:A8
Checking link statuses...
Port 0 Link Up - speed 25000 Mbps - full-duplex
Port 1 Link Up - speed 25000 Mbps - full-duplex
Done
testpmd> 


=> I'll continue with OVS testing on Wednesday--as tomorrow is my travel day back to USA.


In summary:
=> Environment: RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)

[1] Used dpdk-16.11-3.el7.src.rpm
[2] Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
    config/common_linuxapp
[3] Applied patch from:
    - http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415

Comment 56 Andy Gospodarek 2017-12-11 20:34:10 UTC
I'm glad to hear that is working.  Thanks to all for the efforts on this.  We look forward to hearing how performance testing goes.

Comment 57 Somnath Kotur 2017-12-12 03:12:24 UTC
Hi Dilip,
       
> In summary:
> => Environment: RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> 
> [1] Used dpdk-16.11-3.el7.src.rpm
Just to confirm, this is the original src rpm right? Not the handcrafted one i gave/attached in comment# 49 right?

> [2] Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
>     config/common_linuxapp
> [3] Applied patch from:
>     -
> http://www.dpdk.org/browse/dpdk/commit/
> ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415

Yes, really glad to hear it's working now! 

Thanks
Som

Comment 58 Dilip Daya 2017-12-12 05:52:14 UTC
(In reply to Somnath Kotur from comment #57)
> Hi Dilip,
>        
> > In summary:
> > => Environment: RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> > 
> > [1] Used dpdk-16.11-3.el7.src.rpm
> Just to confirm, this is the original src rpm right? Not the handcrafted one
> i gave/attached in comment# 49 right?


Attached to Comment #49
-rw-rw-r-- 1 dilip dilip 9071240 Dec 12 00:45 dpdk-16.11-3.el7.src.rpm
                         ^^^^^^^

I used (IIRC provided by Sandeep Raman):
-rw-r--r--   1 root  root   1051764 Dec  5 11:37 dpdk-16.11-3.el7.x86_64.rpm
-rw-r--r--   1 root  root   9070159 Dec  5 11:37 dpdk-16.11-3.el7.src.rpm
                            ^^^^^^^


 
> > [2] Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
> >     config/common_linuxapp
> > [3] Applied patch from:
> >     -
> > http://www.dpdk.org/browse/dpdk/commit/
> > ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415
> 
> Yes, really glad to hear it's working now! 


Note that I still need to test OVS and will do that tomorrow.  I only tested "testpmd".


Thanks,
-DilipD.

Comment 59 Dilip Daya 2017-12-12 06:06:59 UTC
(In reply to Dilip Daya from comment #58)
> (In reply to Somnath Kotur from comment #57)
> > Hi Dilip,
> >        
> > > In summary:
> > > => Environment: RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> > > 
> > > [1] Used dpdk-16.11-3.el7.src.rpm
> > Just to confirm, this is the original src rpm right? Not the handcrafted one
> > i gave/attached in comment# 49 right?
> 
> 
> Attached to Comment #49
> -rw-rw-r-- 1 dilip dilip 9071240 Dec 12 00:45 dpdk-16.11-3.el7.src.rpm
>                          ^^^^^^^
> 
> I used (IIRC provided by Sandeep Raman):
> -rw-r--r--   1 root  root   1051764 Dec  5 11:37 dpdk-16.11-3.el7.x86_64.rpm
> -rw-r--r--   1 root  root   9070159 Dec  5 11:37 dpdk-16.11-3.el7.src.rpm
>                             ^^^^^^^

Sandeep Raman shared "dpdk-16.11-3.el7.src" with me on Dec-5th via OneDrive.


Thanks,
-DilipD.

Comment 60 Somnath Kotur 2017-12-12 11:40:44 UTC
(In reply to Dilip Daya from comment #58)
> (In reply to Somnath Kotur from comment #57)
> > Hi Dilip,
> >        
> > > In summary:
> > > => Environment: RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> > > 
> > > [1] Used dpdk-16.11-3.el7.src.rpm
> > Just to confirm, this is the original src rpm right? Not the handcrafted one
> > i gave/attached in comment# 49 right?
> 
> 
> Attached to Comment #49
> -rw-rw-r-- 1 dilip dilip 9071240 Dec 12 00:45 dpdk-16.11-3.el7.src.rpm
>                          ^^^^^^^
> 
> I used (IIRC provided by Sandeep Raman):
> -rw-r--r--   1 root  root   1051764 Dec  5 11:37 dpdk-16.11-3.el7.x86_64.rpm
> -rw-r--r--   1 root  root   9070159 Dec  5 11:37 dpdk-16.11-3.el7.src.rpm
>                             ^^^^^^^
> 
> 
>  
> > > [2] Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
> > >     config/common_linuxapp
> > > [3] Applied patch from:
> > >     -
> > > http://www.dpdk.org/browse/dpdk/commit/
> > > ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415
> > 
> > Yes, really glad to hear it's working now! 
> 
> 
> Note that I still need to test OVS and will do that tomorrow.  I only tested
> "testpmd".
Ok sure, although i that you'd reached this point already with some other version? In any case, request you to perform the steps mentioned in comment# 48 for the OVS bring up now

Thanks
Som

Comment 61 Ajit Khaparde 2017-12-13 16:05:04 UTC
FYI..
No need to separately patch the 16.11 code with the PCI IDs anymore.
The patch with the PCI IDs has been applied to the 16.11 stable tree
with commit 6c2d431164f3d837.

----
commit 6c2d431164f3d8374fc9ce4396746495b46cab95 (HEAD, origin/16.11)
Author: Ajit Khaparde <ajit.khaparde@broadcom.com>
Date:   Tue Dec 12 11:39:00 2017 -0800

    net/bnxt: support new PCI IDs
    
    [ upstream commit 1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 ]
    
    Add support for PCI IDs which was removed as a part of
    commit 0e7bd0b2d6a5 ("net/bnxt: remove support for few devices")
    
    Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
    Acked-by: Luca Boccassi <bluca@debian.org>

Comment 62 Dilip Daya 2017-12-14 01:01:58 UTC
On Wed, 2017-12-13 at 16:26 +0000, Sean Merrow wrote:
> Hey Folks,
> 
> Sounds like some good progress is being made with the Broadcom PMD.
> Eventually, you'll want to get this fully supported by Red Hat,
> specifically in OSP 10. 
> 
> You can do the following for the 2.6.1 for OSP10, DPDK 16.11.
> 
> Get the source RPM, for Red Hat OpenStack 10 this is openvswitch-
> 2.6.1-16.git20161206.el7ost.src.rpm
> Install the RPM
> Modify the spec file, and replace the DPDK tar.xz with the DPDK 16.11
> that includes your patches.
> Build the RPM from here, and use it.
> 
> For DPDK 17.11 we do not have an OVS package yet on fast datapath
> (beta)
> 
> NOTE: Getting your patches into DPDK 16.11 upstream is a requirement
> for OSP 10, yet not a guarranttee. You'll have to propose the
> patches, and if they don't make major changes and add too much risk
> to stability of 16.11, then it should get accepted. If it meets that
> criteria, Red Hat engineering will also advocate for the merge in the
> community.
> 
> Thanks,
> Sean
> 
> -- 
> Sean Merrow
> Principal NFV Partner Manager
> Red Hat
> O. 1-978-392-8109
> M. 1-603-321-8663

In Summary my environment thus far:

* RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
* Patched dpdk-16.11.tar.gz as per comment #55 Summary
* openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
  - replaced dpdk-16.11 with patched dpdk-16.11.tar.gz from above.
  - followed steps from comment #48


=> /usr/local/var/log/openvswitch/ovs-vswitchd.log
...
2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:54:50.155Z|00048|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:54:50.155Z|00049|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:54:58.161Z|00050|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:54:58.161Z|00051|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:03.166Z|00052|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:03.166Z|00053|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:08.172Z|00054|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:08.172Z|00055|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:08.175Z|00056|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:08.175Z|00057|bridge|WARN|could not open network device dpdkport0 (No such device)
...


Detailed steps
--------------

1. Get the source RPM, for Red Hat OpenStack 10 this is openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
2. Install the RPM

[root@en104 RH_OVS-DPDK_POC_jason]# pwd
/home/dilip/RH_OVS-DPDK_POC_jason

[root@en104 RH_OVS-DPDK_POC_jason]# rpm -Uvh openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
Updating / installing...
   1:openvswitch-2.6.1-16.git20161206.################################# [100%]


3. Modify the spec file, and replace the DPDK tar.xz with the DPDK 16.11 that includes your patches.

(a) [root@en104 SPECS]# pwd
/root/rpmbuild/SPECS

[root@en104 SPECS]# grep BNXT *
setconf CONFIG_RTE_LIBRTE_BNXT_PMD n

<<< modify spec file >>>

[root@en104 SPECS]# grep BNXT *
setconf CONFIG_RTE_LIBRTE_BNXT_PMD y


(b) Replaced dpdk-16.11.tar.gz with patched dpdk-16.11.tar.gz :

[root@en104 Dec-5-Sandeep-working]# pwd
/home/dilip/RH_OVS-DPDK_POC_jason/Dec-5-Sandeep-working

[root@en104 Dec-5-Sandeep-working]# l
total 30020
-rw-r--r--  1 root root  1051764 Dec  5 11:37 dpdk-16.11-3.el7.x86_64.rpm
-rw-r--r--  1 root root  9070159 Dec  5 11:37 dpdk-16.11-3.el7.src.rpm
drwxr-xr-x 13 root root      298 Dec 13 11:43 dpdk-16.11/
-rw-r--r--  1 root root     3620 Dec 13 11:47 dpdk-16.11-3.el7_631SFP28.patch
-rw-r--r--  1 root root  9739452 Dec 13 11:50 dpdk-16.11_patched.tar.xz
-rw-r--r--  1 root root     3095 Dec 13 11:51 add_support_for_new_pci_ids.patch
-rw-r--r--  1 root root 10864569 Dec 13 12:01 dpdk-16.11_patched.tar.gz

Copied dpdk-16.11_patched.tar.gz to /root/rpmbuild/SOURCES/dpdk-16.11.tar.gz
(dpdk-16.11_patched.tar.gz as per comment #55 Summary)

4. Build the RPM from here, and use it.

Followed steps from:
- https://bugzilla.redhat.com/show_bug.cgi?id=1514088#c48

[root@en104 Dec-5-Sandeep-working]# pwd
/home/dilip/RH_OVS-DPDK_POC_jason/Dec-5-Sandeep-working
[root@en104 Dec-5-Sandeep-working]# ./show_dpdk_status 
+ DPDK=/root/rpmbuild_dpdk-16.11-3.el7-Dec-5-working/SOURCES/dpdk-16.11
+ BINDER=tools/dpdk-devbind.py
+ /root/rpmbuild_dpdk-16.11-3.el7-Dec-5-working/SOURCES/dpdk-16.11/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


[root@en104 Dec-5-Sandeep-working]# ./start_ovs_and_looped_flows_no_comments.sh 
++ export OVSSCHEMA=/usr/local/share/openvswitch/vswitch.ovsschema
++ OVSSCHEMA=/usr/local/share/openvswitch/vswitch.ovsschema
++ export OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
++ OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
++ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
++ P1=0000:12:00.0
++ P2=0000:12:00.1
++ ovs-ctl stop
ovs-vswitchd is not running.
ovsdb-server is not running.
++ pkill -9 ovs
++ rm -rf /usr/local/var/run/openvswitch
++ rm -rf /usr/local/etc/openvswitch/
++ rm -rf /usr/local/etc/openvswitch/conf.db
++ rm -fr /usr/local/var/log/openvswitch/
++ mkdir -p /usr/local/etc/openvswitch
++ mkdir -p /usr/local/var/run/openvswitch
++ mkdir -p /usr/local/var/log/openvswitch
++ modprobe openvswitch
++ ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema
++ sleep 6
++ ovs-ctl --system-id=random --no-ovs-vswitchd start
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=1
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
++ sleep 6
++ ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
++ sleep 6
++ ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7f6c40000000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7f6c40112000M
EAL: PCI device 0000:5d:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL: PCI device 0000:5d:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100, virt:0x7f6c3ffced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
++ sleep 6
++ ovs-ctl status
ovsdb-server is running with pid 5978
ovs-vswitchd is running with pid 6059
++ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport0 -- set Interface dpdkport0 type=dpdk options:dpdk-devargs=0000:12:00.0 ofport_request=1
ovs-vsctl: Error detected while setting up 'dpdkport0'.  See ovs-vswitchd log for details.
++ sleep 8
++ ovs-vsctl add-port br0 dpdkport1 -- set Interface dpdkport1 type=dpdk options:dpdk-devargs=0000:12:00.1 ofport_request=2
ovs-vsctl: Error detected while setting up 'dpdkport1'.  See ovs-vswitchd log for details.
++ sleep 8
++ ovs-vsctl set Interface dpdkport0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
++ sleep 5
++ ovs-vsctl set Interface dpdkport1 options:n_rxq=1 other_config:pmd-rxq-affinity=0:2
++ sleep 5
++ ovs-vsctl set Interface dpdkport0 options:n_rxq_desc=2048
++ ovs-vsctl set Interface dpdkport1 options:n_rxq_desc=2048
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=34.035s, table=0, n_packets=0, n_bytes=0, idle_age=34, priority=0 actions=NORMAL
++ ovs-ofctl del-flows br0
++ sleep 5
++ ovs-ofctl add-flow br0 in_port=1,action=output:2
++ ovs-ofctl add-flow br0 in_port=2,action=output:1
++ ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=0.003s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=2 actions=output:1
++ ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000f2942f28614e
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 LOCAL(br0): addr:f2:94:2f:28:61:4e
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
++ ovs-appctl dpif-netdev/pmd-stats-show
main thread:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:573366 (100.00%)
	processing cycles:0 (0.00%)
++ ovs-appctl dpif-netdev/pmd-rxq-show
++ ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 1 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
++ ovs-vsctl list interface
_uuid               : 1ced6384-3561-44a8-9549-325356682795
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "could not open network device dpdkport1 (No such device)"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport1"
ofport              : -1
ofport_request      : 2
options             : {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:2"}
statistics          : {}
status              : {}
type                : dpdk

_uuid               : 87c3dba9-9aa8-4610-a50e-40d0aa64ee28
admin_state         : []
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : "could not open network device dpdkport0 (No such device)"
external_ids        : {}
ifindex             : []
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : []
link_speed          : []
link_state          : []
lldp                : {}
mac                 : []
mac_in_use          : []
mtu                 : []
mtu_request         : []
name                : "dpdkport0"
ofport              : -1
ofport_request      : 1
options             : {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {}
status              : {}
type                : dpdk

_uuid               : b937f88b-1322-4dd0-a63a-96aa5486c234
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 19
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "f2:94:2f:28:61:4e"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal
++ ps -ef
++ grep ovs
root      5977     1  0 13:53 ?        00:00:00 ovsdb-server: monitoring pid 5978 (healthy)
root      5978  5977  0 13:53 ?        00:00:00 ovsdb-server /usr/local/etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/usr/local/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root      6058     1  0 13:53 ?        00:00:00 ovs-vswitchd: monitoring pid 6059 (healthy)
root      6059  6058 46 13:53 ?        00:00:39 ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor



[root@en104 Dec-5-Sandeep-working]# cat /usr/local/var/log/openvswitch/ovsdb-server.log
2017-12-13T20:53:11.737Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovsdb-server.log
2017-12-13T20:53:11.741Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.6.2
2017-12-13T20:53:21.764Z|00003|memory|INFO|2284 kB peak resident set size after 10.0 seconds
2017-12-13T20:53:21.764Z|00004|memory|INFO|cells:16 monitors:0



[root@en104 Dec-5-Sandeep-working]# cat /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-13T20:53:47.796Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-13T20:53:47.800Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0
2017-12-13T20:53:47.800Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1
2017-12-13T20:53:47.800Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores
2017-12-13T20:53:47.800Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
2017-12-13T20:53:47.800Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
2017-12-13T20:53:47.801Z|00007|dpdk|INFO|DPDK Enabled, initializing
2017-12-13T20:53:47.801Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /usr/local/var/run/openvswitch
2017-12-13T20:53:47.801Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1 --socket-mem 2048,0
2017-12-13T20:54:28.101Z|00010|timeval|WARN|Unreasonably long 40300ms poll interval (34ms user, 39668ms system)
2017-12-13T20:54:28.101Z|00011|timeval|WARN|faults: 81067 minor, 0 major
2017-12-13T20:54:28.101Z|00012|timeval|WARN|context switches: 5 voluntary, 28 involuntary
2017-12-13T20:54:28.101Z|00013|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=fde89638:
2017-12-13T20:54:28.101Z|00014|coverage|INFO|bridge_reconfigure         0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00015|coverage|INFO|cmap_expand                0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-13T20:54:28.101Z|00016|coverage|INFO|miniflow_malloc            0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-13T20:54:28.101Z|00017|coverage|INFO|hmap_pathological          0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00018|coverage|INFO|hmap_expand               16.2/sec    10.800/sec        0.1800/sec   total: 648
2017-12-13T20:54:28.101Z|00019|coverage|INFO|txn_unchanged              0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-13T20:54:28.101Z|00020|coverage|INFO|txn_incomplete             0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00021|coverage|INFO|poll_create_node           1.4/sec     0.700/sec        0.0117/sec   total: 42
2017-12-13T20:54:28.101Z|00022|coverage|INFO|seq_change                 1.8/sec     0.733/sec        0.0122/sec   total: 44
2017-12-13T20:54:28.101Z|00023|coverage|INFO|pstream_open               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00024|coverage|INFO|stream_open                0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00025|coverage|INFO|util_xalloc              282.2/sec   187.317/sec        3.1219/sec   total: 11239
2017-12-13T20:54:28.101Z|00026|coverage|INFO|netdev_get_hwaddr          0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-13T20:54:28.101Z|00027|coverage|INFO|netlink_received           0.6/sec     0.050/sec        0.0008/sec   total: 3
2017-12-13T20:54:28.101Z|00028|coverage|INFO|netlink_sent               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-13T20:54:28.101Z|00029|coverage|INFO|88 events never hit
2017-12-13T20:54:28.103Z|00030|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98% CPU usage)
2017-12-13T20:54:28.103Z|00031|memory|INFO|59264 kB peak resident set size after 40.3 seconds
2017-12-13T20:54:28.103Z|00032|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.2
2017-12-13T20:54:34.134Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-12-13T20:54:34.134Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-12-13T20:54:34.134Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action
2017-12-13T20:54:34.134Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-12-13T20:54:34.134Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state
2017-12-13T20:54:34.134Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone
2017-12-13T20:54:34.134Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark
2017-12-13T20:54:34.134Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label
2017-12-13T20:54:34.134Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state_nat
2017-12-13T20:54:34.142Z|00042|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-13T20:54:34.142Z|00043|bridge|INFO|bridge br0: using datapath ID 0000f2942f28614e
2017-12-13T20:54:34.142Z|00044|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-13T20:54:38.152Z|00045|memory|INFO|peak resident set size grew 54% in last 10.1 seconds, from 59264 kB to 91148 kB
2017-12-13T20:54:38.152Z|00046|memory|INFO|handlers:1 ports:1 revalidators:1 rules:5
2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:54:50.155Z|00048|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:54:50.155Z|00049|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:54:58.161Z|00050|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:54:58.161Z|00051|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:03.166Z|00052|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:03.166Z|00053|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:08.172Z|00054|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:08.172Z|00055|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:08.175Z|00056|bridge|WARN|could not open network device dpdkport1 (No such device)
2017-12-13T20:55:08.175Z|00057|bridge|WARN|could not open network device dpdkport0 (No such device)
2017-12-13T20:55:08.179Z|00058|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 deletes)
2017-12-13T20:55:13.187Z|00059|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-12-13T20:55:13.188Z|00060|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)

Comment 63 Somnath Kotur 2017-12-14 03:57:41 UTC
(In reply to Dilip Daya from comment #62)
> On Wed, 2017-12-13 at 16:26 +0000, Sean Merrow wrote:
> > Hey Folks,
> > 
> > Sounds like some good progress is being made with the Broadcom PMD.
> > Eventually, you'll want to get this fully supported by Red Hat,
> > specifically in OSP 10. 
> > 
> > You can do the following for the 2.6.1 for OSP10, DPDK 16.11.
> > 
> > Get the source RPM, for Red Hat OpenStack 10 this is openvswitch-
> > 2.6.1-16.git20161206.el7ost.src.rpm
> > Install the RPM
> > Modify the spec file, and replace the DPDK tar.xz with the DPDK 16.11
> > that includes your patches.
> > Build the RPM from here, and use it.
> > 
> > For DPDK 17.11 we do not have an OVS package yet on fast datapath
> > (beta)
> > 
> > NOTE: Getting your patches into DPDK 16.11 upstream is a requirement
> > for OSP 10, yet not a guarranttee. You'll have to propose the
> > patches, and if they don't make major changes and add too much risk
> > to stability of 16.11, then it should get accepted. If it meets that
> > criteria, Red Hat engineering will also advocate for the merge in the
> > community.
> > 
> > Thanks,
> > Sean
> > 
> > -- 
> > Sean Merrow
> > Principal NFV Partner Manager
> > Red Hat
> > O. 1-978-392-8109
> > M. 1-603-321-8663
> 
> In Summary my environment thus far:
> 
> * RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> * Patched dpdk-16.11.tar.gz as per comment #55 Summary
> * openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
>   - replaced dpdk-16.11 with patched dpdk-16.11.tar.gz from above.
>   - followed steps from comment #48
> 
> 
> => /usr/local/var/log/openvswitch/ovs-vswitchd.log
> ...
> 2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:54:50.155Z|00048|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:54:50.155Z|00049|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:54:58.161Z|00050|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:54:58.161Z|00051|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:03.166Z|00052|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:03.166Z|00053|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:08.172Z|00054|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:08.172Z|00055|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:08.175Z|00056|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:08.175Z|00057|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> ...
> 
> 
> Detailed steps
> --------------
> 
> 1. Get the source RPM, for Red Hat OpenStack 10 this is
> openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
> 2. Install the RPM
> 
> [root@en104 RH_OVS-DPDK_POC_jason]# pwd
> /home/dilip/RH_OVS-DPDK_POC_jason
> 
> [root@en104 RH_OVS-DPDK_POC_jason]# rpm -Uvh
> openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
> Updating / installing...
>    1:openvswitch-2.6.1-16.git20161206.#################################
> [100%]
> 
> 
> 3. Modify the spec file, and replace the DPDK tar.xz with the DPDK 16.11
> that includes your patches.
> 
> (a) [root@en104 SPECS]# pwd
> /root/rpmbuild/SPECS
> 
> [root@en104 SPECS]# grep BNXT *
> setconf CONFIG_RTE_LIBRTE_BNXT_PMD n
> 
> <<< modify spec file >>>
> 
> [root@en104 SPECS]# grep BNXT *
> setconf CONFIG_RTE_LIBRTE_BNXT_PMD y
> 
> 
> (b) Replaced dpdk-16.11.tar.gz with patched dpdk-16.11.tar.gz :
> 
> [root@en104 Dec-5-Sandeep-working]# pwd
> /home/dilip/RH_OVS-DPDK_POC_jason/Dec-5-Sandeep-working
> 
> [root@en104 Dec-5-Sandeep-working]# l
> total 30020
> -rw-r--r--  1 root root  1051764 Dec  5 11:37 dpdk-16.11-3.el7.x86_64.rpm
> -rw-r--r--  1 root root  9070159 Dec  5 11:37 dpdk-16.11-3.el7.src.rpm
> drwxr-xr-x 13 root root      298 Dec 13 11:43 dpdk-16.11/
> -rw-r--r--  1 root root     3620 Dec 13 11:47 dpdk-16.11-3.el7_631SFP28.patch
> -rw-r--r--  1 root root  9739452 Dec 13 11:50 dpdk-16.11_patched.tar.xz
> -rw-r--r--  1 root root     3095 Dec 13 11:51
> add_support_for_new_pci_ids.patch
> -rw-r--r--  1 root root 10864569 Dec 13 12:01 dpdk-16.11_patched.tar.gz
> 
> Copied dpdk-16.11_patched.tar.gz to /root/rpmbuild/SOURCES/dpdk-16.11.tar.gz
> (dpdk-16.11_patched.tar.gz as per comment #55 Summary)
> 
> 4. Build the RPM from here, and use it.
> 
> Followed steps from:
> - https://bugzilla.redhat.com/show_bug.cgi?id=1514088#c48
> 
> [root@en104 Dec-5-Sandeep-working]# pwd
> /home/dilip/RH_OVS-DPDK_POC_jason/Dec-5-Sandeep-working
> [root@en104 Dec-5-Sandeep-working]# ./show_dpdk_status 
> + DPDK=/root/rpmbuild_dpdk-16.11-3.el7-Dec-5-working/SOURCES/dpdk-16.11
> + BINDER=tools/dpdk-devbind.py
> +
> /root/rpmbuild_dpdk-16.11-3.el7-Dec-5-working/SOURCES/dpdk-16.11/tools/dpdk-
> devbind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 
> 
> [root@en104 Dec-5-Sandeep-working]#
> ./start_ovs_and_looped_flows_no_comments.sh 
> ++ export OVSSCHEMA=/usr/local/share/openvswitch/vswitch.ovsschema
> ++ OVSSCHEMA=/usr/local/share/openvswitch/vswitch.ovsschema
> ++ export OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
> ++ OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
> ++ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> ++ DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> ++ export
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/
> share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
> ++
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/
> share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
> ++ P1=0000:12:00.0
> ++ P2=0000:12:00.1
> ++ ovs-ctl stop
> ovs-vswitchd is not running.
> ovsdb-server is not running.
> ++ pkill -9 ovs
> ++ rm -rf /usr/local/var/run/openvswitch
> ++ rm -rf /usr/local/etc/openvswitch/
> ++ rm -rf /usr/local/etc/openvswitch/conf.db
> ++ rm -fr /usr/local/var/log/openvswitch/
> ++ mkdir -p /usr/local/etc/openvswitch
> ++ mkdir -p /usr/local/var/run/openvswitch
> ++ mkdir -p /usr/local/var/log/openvswitch
> ++ modprobe openvswitch
> ++ ovsdb-tool create /usr/local/etc/openvswitch/conf.db
> /usr/local/share/openvswitch/vswitch.ovsschema
> ++ sleep 6
> ++ ovs-ctl --system-id=random --no-ovs-vswitchd start
> Starting ovsdb-server                                      [  OK  ]
> Configuring Open vSwitch system IDs                        [  OK  ]
> ++ sleep 6
> ++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=1
> ++ sleep 6
> ++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
> ++ sleep 6
> ++ ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6
> ++ sleep 6
> ++ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
> ++ sleep 6
> ++ ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
> ++ sleep 6
> ++ ovs-ctl --no-ovsdb-server
> --db-sock=/usr/local/var/run/openvswitch/db.sock start
> Starting ovs-vswitchd EAL: Detected 32 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:12:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL:   using IOMMU type 1 (Type 1)
> PMD: Broadcom Cumulus driver bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da600000, node addr 0x7f6c40000000M
> EAL: PCI device 0000:12:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da610000, node addr 0x7f6c40112000M
> EAL: PCI device 0000:5d:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL: PCI device 0000:5d:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100,
> virt:0x7f6c3ffced40, socket_id:0, flags:0
>                                                            [  OK  ]
> Enabling remote OVSDB managers                             [  OK  ]
> ++ sleep 6
> ++ ovs-ctl status
> ovsdb-server is running with pid 5978
> ovs-vswitchd is running with pid 6059
> ++ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> ++ sleep 8
> ++ ovs-vsctl add-port br0 dpdkport0 -- set Interface dpdkport0 type=dpdk
> options:dpdk-devargs=0000:12:00.0 ofport_request=1
> ovs-vsctl: Error detected while setting up 'dpdkport0'.  See ovs-vswitchd
> log for details.
> ++ sleep 8
> ++ ovs-vsctl add-port br0 dpdkport1 -- set Interface dpdkport1 type=dpdk
> options:dpdk-devargs=0000:12:00.1 ofport_request=2
> ovs-vsctl: Error detected while setting up 'dpdkport1'.  See ovs-vswitchd
> log for details.
> ++ sleep 8
> ++ ovs-vsctl set Interface dpdkport0 options:n_rxq=1
> other_config:pmd-rxq-affinity=0:1
> ++ sleep 5
> ++ ovs-vsctl set Interface dpdkport1 options:n_rxq=1
> other_config:pmd-rxq-affinity=0:2
> ++ sleep 5
> ++ ovs-vsctl set Interface dpdkport0 options:n_rxq_desc=2048
> ++ ovs-vsctl set Interface dpdkport1 options:n_rxq_desc=2048
> ++ ovs-ofctl dump-flows br0
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=34.035s, table=0, n_packets=0, n_bytes=0, idle_age=34,
> priority=0 actions=NORMAL
> ++ ovs-ofctl del-flows br0
> ++ sleep 5
> ++ ovs-ofctl add-flow br0 in_port=1,action=output:2
> ++ ovs-ofctl add-flow br0 in_port=2,action=output:1
> ++ ovs-ofctl dump-flows br0
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=0.003s, table=0, n_packets=0, n_bytes=0, idle_age=0,
> in_port=1 actions=output:2
>  cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0,
> in_port=2 actions=output:1
> ++ ovs-ofctl show br0
> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000f2942f28614e
> n_tables:254, n_buffers:256
> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
> actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src
> mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
>  LOCAL(br0): addr:f2:94:2f:28:61:4e
>      config:     PORT_DOWN
>      state:      LINK_DOWN
>      current:    10MB-FD COPPER
>      speed: 10 Mbps now, 0 Mbps max
> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
> ++ ovs-appctl dpif-netdev/pmd-stats-show
> main thread:
> 	emc hits:0
> 	megaflow hits:0
> 	avg. subtable lookups per hit:0.00
> 	miss:0
> 	lost:0
> 	polling cycles:573366 (100.00%)
> 	processing cycles:0 (0.00%)
> ++ ovs-appctl dpif-netdev/pmd-rxq-show
> ++ ovs-ofctl dump-ports br0
> OFPST_PORT reply (xid=0x2): 1 ports
>   port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
>            tx pkts=0, bytes=0, drop=0, errs=0, coll=0
> ++ ovs-vsctl list interface
> _uuid               : 1ced6384-3561-44a8-9549-325356682795
> admin_state         : []
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : []
> error               : "could not open network device dpdkport1 (No such
> device)"
> external_ids        : {}
> ifindex             : []
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : []
> link_speed          : []
> link_state          : []
> lldp                : {}
> mac                 : []
> mac_in_use          : []
> mtu                 : []
> mtu_request         : []
> name                : "dpdkport1"
> ofport              : -1
> ofport_request      : 2
> options             : {dpdk-devargs="0000:12:00.1", n_rxq="1",
> n_rxq_desc="2048"}
> other_config        : {pmd-rxq-affinity="0:2"}
> statistics          : {}
> status              : {}
> type                : dpdk
> 
> _uuid               : 87c3dba9-9aa8-4610-a50e-40d0aa64ee28
> admin_state         : []
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : []
> error               : "could not open network device dpdkport0 (No such
> device)"
> external_ids        : {}
> ifindex             : []
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : []
> link_speed          : []
> link_state          : []
> lldp                : {}
> mac                 : []
> mac_in_use          : []
> mtu                 : []
> mtu_request         : []
> name                : "dpdkport0"
> ofport              : -1
> ofport_request      : 1
> options             : {dpdk-devargs="0000:12:00.0", n_rxq="1",
> n_rxq_desc="2048"}
> other_config        : {pmd-rxq-affinity="0:1"}
> statistics          : {}
> status              : {}
> type                : dpdk
> 
> _uuid               : b937f88b-1322-4dd0-a63a-96aa5486c234
> admin_state         : down
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : full
> error               : []
> external_ids        : {}
> ifindex             : 19
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : 0
> link_speed          : 10000000
> link_state          : down
> lldp                : {}
> mac                 : []
> mac_in_use          : "f2:94:2f:28:61:4e"
> mtu                 : 1500
> mtu_request         : []
> name                : "br0"
> ofport              : 65534
> ofport_request      : []
> options             : {}
> other_config        : {}
> statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0,
> rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0,
> tx_dropped=0, tx_errors=0, tx_packets=0}
> status              : {driver_name=tun, driver_version="1.6",
> firmware_version=""}
> type                : internal
> ++ ps -ef
> ++ grep ovs
> root      5977     1  0 13:53 ?        00:00:00 ovsdb-server: monitoring pid
> 5978 (healthy)
> root      5978  5977  0 13:53 ?        00:00:00 ovsdb-server
> /usr/local/etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info
> --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
> --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log
> --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid --detach --monitor
> root      6058     1  0 13:53 ?        00:00:00 ovs-vswitchd: monitoring pid
> 6059 (healthy)
> root      6059  6058 46 13:53 ?        00:00:39 ovs-vswitchd
> unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
> -vfile:info --mlockall --no-chdir
> --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
> 
> 
> 
> [root@en104 Dec-5-Sandeep-working]# cat
> /usr/local/var/log/openvswitch/ovsdb-server.log
> 2017-12-13T20:53:11.737Z|00001|vlog|INFO|opened log file
> /usr/local/var/log/openvswitch/ovsdb-server.log
> 2017-12-13T20:53:11.741Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch)
> 2.6.2
> 2017-12-13T20:53:21.764Z|00003|memory|INFO|2284 kB peak resident set size
> after 10.0 seconds
> 2017-12-13T20:53:21.764Z|00004|memory|INFO|cells:16 monitors:0
> 
> 
> 
> [root@en104 Dec-5-Sandeep-working]# cat
> /usr/local/var/log/openvswitch/ovs-vswitchd.log
> 2017-12-13T20:53:47.796Z|00001|vlog|INFO|opened log file
> /usr/local/var/log/openvswitch/ovs-vswitchd.log
> 2017-12-13T20:53:47.800Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA
> node 0
> 2017-12-13T20:53:47.800Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA
> node 1
> 2017-12-13T20:53:47.800Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32
> CPU cores
> 2017-12-13T20:53:47.800Z|00005|reconnect|INFO|unix:/usr/local/var/run/
> openvswitch/db.sock: connecting...
> 2017-12-13T20:53:47.800Z|00006|reconnect|INFO|unix:/usr/local/var/run/
> openvswitch/db.sock: connected
> 2017-12-13T20:53:47.801Z|00007|dpdk|INFO|DPDK Enabled, initializing
> 2017-12-13T20:53:47.801Z|00008|dpdk|INFO|No vhost-sock-dir provided -
> defaulting to /usr/local/var/run/openvswitch
> 2017-12-13T20:53:47.801Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1
> --socket-mem 2048,0
> 2017-12-13T20:54:28.101Z|00010|timeval|WARN|Unreasonably long 40300ms poll
> interval (34ms user, 39668ms system)
> 2017-12-13T20:54:28.101Z|00011|timeval|WARN|faults: 81067 minor, 0 major
> 2017-12-13T20:54:28.101Z|00012|timeval|WARN|context switches: 5 voluntary,
> 28 involuntary
> 2017-12-13T20:54:28.101Z|00013|coverage|INFO|Event coverage, avg rate over
> last: 5 seconds, last minute, last hour,  hash=fde89638:
> 2017-12-13T20:54:28.101Z|00014|coverage|INFO|bridge_reconfigure        
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00015|coverage|INFO|cmap_expand               
> 0.4/sec     0.150/sec        0.0025/sec   total: 9
> 2017-12-13T20:54:28.101Z|00016|coverage|INFO|miniflow_malloc           
> 0.4/sec     0.150/sec        0.0025/sec   total: 9
> 2017-12-13T20:54:28.101Z|00017|coverage|INFO|hmap_pathological         
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00018|coverage|INFO|hmap_expand              
> 16.2/sec    10.800/sec        0.1800/sec   total: 648
> 2017-12-13T20:54:28.101Z|00019|coverage|INFO|txn_unchanged             
> 0.4/sec     0.033/sec        0.0006/sec   total: 2
> 2017-12-13T20:54:28.101Z|00020|coverage|INFO|txn_incomplete            
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00021|coverage|INFO|poll_create_node          
> 1.4/sec     0.700/sec        0.0117/sec   total: 42
> 2017-12-13T20:54:28.101Z|00022|coverage|INFO|seq_change                
> 1.8/sec     0.733/sec        0.0122/sec   total: 44
> 2017-12-13T20:54:28.101Z|00023|coverage|INFO|pstream_open              
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00024|coverage|INFO|stream_open               
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00025|coverage|INFO|util_xalloc             
> 282.2/sec   187.317/sec        3.1219/sec   total: 11239
> 2017-12-13T20:54:28.101Z|00026|coverage|INFO|netdev_get_hwaddr         
> 0.4/sec     0.033/sec        0.0006/sec   total: 2
> 2017-12-13T20:54:28.101Z|00027|coverage|INFO|netlink_received          
> 0.6/sec     0.050/sec        0.0008/sec   total: 3
> 2017-12-13T20:54:28.101Z|00028|coverage|INFO|netlink_sent              
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-13T20:54:28.101Z|00029|coverage|INFO|88 events never hit
> 2017-12-13T20:54:28.103Z|00030|poll_loop|INFO|wakeup due to [POLLIN] on fd
> 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98%
> CPU usage)
> 2017-12-13T20:54:28.103Z|00031|memory|INFO|59264 kB peak resident set size
> after 40.3 seconds
> 2017-12-13T20:54:28.103Z|00032|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.2
> 2017-12-13T20:54:34.134Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports recirculation
> 2017-12-13T20:54:34.134Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
> label stack length probed as 3
> 2017-12-13T20:54:34.134Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports truncate action
> 2017-12-13T20:54:34.134Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports unique flow ids
> 2017-12-13T20:54:34.134Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_state
> 2017-12-13T20:54:34.134Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_zone
> 2017-12-13T20:54:34.134Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_mark
> 2017-12-13T20:54:34.134Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_label
> 2017-12-13T20:54:34.134Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> does not support ct_state_nat
> 2017-12-13T20:54:34.142Z|00042|bridge|INFO|bridge br0: added interface br0
> on port 65534
> 2017-12-13T20:54:34.142Z|00043|bridge|INFO|bridge br0: using datapath ID
> 0000f2942f28614e
> 2017-12-13T20:54:34.142Z|00044|connmgr|INFO|br0: added service controller
> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
> 2017-12-13T20:54:38.152Z|00045|memory|INFO|peak resident set size grew 54%
> in last 10.1 seconds, from 59264 kB to 91148 kB
> 2017-12-13T20:54:38.152Z|00046|memory|INFO|handlers:1 ports:1 revalidators:1
> rules:5
> 2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:54:50.155Z|00048|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:54:50.155Z|00049|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:54:58.161Z|00050|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:54:58.161Z|00051|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:03.166Z|00052|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:03.166Z|00053|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:08.172Z|00054|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:08.172Z|00055|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:08.175Z|00056|bridge|WARN|could not open network device
> dpdkport1 (No such device)
> 2017-12-13T20:55:08.175Z|00057|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 2017-12-13T20:55:08.179Z|00058|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 deletes)
> 2017-12-13T20:55:13.187Z|00059|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 adds)
> 2017-12-13T20:55:13.188Z|00060|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 adds)

Hi Dilip,
          Would have been nice if you had followed my steps as mentioned in comment# 48 as is just so we are on the same page.. 
Having said that from what i see in your log, there is definite progress in the sense that OVS also is now detecting the bnxt ports (pasted below) which i don't believe you'd seen before ?
> Starting ovs-vswitchd EAL: Detected 32 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:12:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL:   using IOMMU type 1 (Type 1)
> PMD: Broadcom Cumulus driver bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da600000, node addr 0x7f6c40000000M
> EAL: PCI device 0000:12:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da610000, node addr 0x7f6c40112000M
> EAL: PCI device 0000:5d:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL: PCI device 0000:5d:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100,
> virt:0x7f6c3ffced40, socket_id:0, flags:0
>                                                            [  OK  ]
> Enabling remote OVSDB managers                             [  OK  ]

Now, the only diff i can see between your steps and mine is that it appears as the way you have started your configuration database - the ovsdb-server
11)ovsdb-tool create /usr/local/etc/openvswitch/conf.db oswitchd/vswitch.ovsschema
12) rm -rf /usr/local/var/run/openvswitch
13) mkdir -p /usr/local/var/run/openvswitch
14) ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file
15) export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

and the subsequent initialization of this database like so:
16) ovs-vsctl --no-wait init

The only step i do for DPDK is this (below) before starting off the ovs-vswitchd
17) ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

Although like i said, the OVS has this time successfully detected the BNXT PMD and the error that you see seems OVS specific now

2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device dpdkport0 (No such device)

A quick search yielded a known RH bug and a workaround for this symptom 

https://bugzilla.redhat.com/show_bug.cgi?id=1397299

You might want to try this as well ?

Ajit,
      Could you also please set this up on your systems so you could guide Dilip runtime in his timezone ?

Thanks
Som

Comment 64 Aaron Conole 2017-12-14 15:08:04 UTC
I've spotted an error with the 2.6 setup.  You have named your port dpdkport0 and used dpdk-devargs.  However, that is only added as of OvS 2.7.  Try with dpdk{0,1,2,...) and do not use dpdk-devargs.

Comment 65 Dilip Daya 2017-12-14 15:35:25 UTC
(In reply to Aaron Conole from comment #64)
> I've spotted an error with the 2.6 setup.  You have named your port
> dpdkport0 and used dpdk-devargs.  However, that is only added as of OvS 2.7.
> Try with dpdk{0,1,2,...) and do not use dpdk-devargs.


[root@en104 Dec-5-Sandeep-working]# ./start_ovs_and_looped_flows_no_comments_git20161206.sh
++ export OVSSCHEMA=/root/rpmbuild/BUILD/openvswitch-2.6.1/vswitchd/vswitch.ovsschema
++ OVSSCHEMA=/root/rpmbuild/BUILD/openvswitch-2.6.1/vswitchd/vswitch.ovsschema
++ export OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
++ OVS_DIR=/root/rpmbuild/BUILD/openvswitch-2.6.1
++ export OVS_UTILS=/root/rpmbuild/BUILD/openvswitch-2.6.1/utilities
++ OVS_UTILS=/root/rpmbuild/BUILD/openvswitch-2.6.1/utilities
++ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts
++ P1=0000:12:00.0
++ P2=0000:12:00.1
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl stop
ovs-vswitchd is not running.
ovsdb-server is not running.
++ pkill -9 ovs
++ rm -rf /usr/local/var/run/openvswitch
++ rm -rf /usr/local/etc/openvswitch/
++ rm -rf /usr/local/etc/openvswitch/conf.db
++ rm -fr /usr/local/var/log/openvswitch/
++ mkdir -p /usr/local/etc/openvswitch
++ mkdir -p /usr/local/var/run/openvswitch
++ mkdir -p /usr/local/var/log/openvswitch
++ modprobe openvswitch
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db /root/rpmbuild/BUILD/openvswitch-2.6.1/vswitchd/vswitch.ovsschema
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl --system-id=random --no-ovs-vswitchd start
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=1
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7f9ac0000000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7f9ac0112000M
EAL: PCI device 0000:5d:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL: PCI device 0000:5d:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100, virt:0x7f9abffced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
++ sleep 6
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl status
ovsdb-server is running with pid 10007
ovs-vswitchd is running with pid 10095
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
++ sleep 8
++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.



[root@en104 Dec-5-Sandeep-working]# cat /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-14T15:31:52.241Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-14T15:31:52.245Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0
2017-12-14T15:31:52.245Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1
2017-12-14T15:31:52.245Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores
2017-12-14T15:31:52.245Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
2017-12-14T15:31:52.245Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
2017-12-14T15:31:52.246Z|00007|dpdk|INFO|DPDK Enabled, initializing
2017-12-14T15:31:52.246Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /usr/local/var/run/openvswitch
2017-12-14T15:31:52.246Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1 --socket-mem 2048,0
2017-12-14T15:32:32.809Z|00010|timeval|WARN|Unreasonably long 40562ms poll interval (48ms user, 39916ms system)
2017-12-14T15:32:32.809Z|00011|timeval|WARN|faults: 81006 minor, 0 major
2017-12-14T15:32:32.809Z|00012|timeval|WARN|context switches: 5 voluntary, 48 involuntary
2017-12-14T15:32:32.809Z|00013|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=fde89638:
2017-12-14T15:32:32.809Z|00014|coverage|INFO|bridge_reconfigure         0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00015|coverage|INFO|cmap_expand                0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:32:32.809Z|00016|coverage|INFO|miniflow_malloc            0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:32:32.809Z|00017|coverage|INFO|hmap_pathological          0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00018|coverage|INFO|hmap_expand               16.2/sec    10.800/sec        0.1800/sec   total: 648
2017-12-14T15:32:32.809Z|00019|coverage|INFO|txn_unchanged              0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:32:32.809Z|00020|coverage|INFO|txn_incomplete             0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00021|coverage|INFO|poll_create_node           1.4/sec     0.700/sec        0.0117/sec   total: 42
2017-12-14T15:32:32.809Z|00022|coverage|INFO|seq_change                 1.8/sec     0.733/sec        0.0122/sec   total: 44
2017-12-14T15:32:32.809Z|00023|coverage|INFO|pstream_open               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00024|coverage|INFO|stream_open                0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00025|coverage|INFO|util_xalloc              282.2/sec   187.317/sec        3.1219/sec   total: 11239
2017-12-14T15:32:32.809Z|00026|coverage|INFO|netdev_get_hwaddr          0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:32:32.809Z|00027|coverage|INFO|netlink_received           0.6/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:32:32.809Z|00028|coverage|INFO|netlink_sent               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:32.809Z|00029|coverage|INFO|88 events never hit
2017-12-14T15:32:32.811Z|00030|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98% CPU usage)
2017-12-14T15:32:32.811Z|00031|memory|INFO|59268 kB peak resident set size after 40.6 seconds
2017-12-14T15:32:32.811Z|00032|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.2
2017-12-14T15:32:38.853Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-12-14T15:32:38.854Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-12-14T15:32:38.854Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action
2017-12-14T15:32:38.854Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-12-14T15:32:38.854Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state
2017-12-14T15:32:38.854Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone
2017-12-14T15:32:38.854Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark
2017-12-14T15:32:38.854Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label
2017-12-14T15:32:38.854Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state_nat
2017-12-14T15:32:38.862Z|00042|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-14T15:32:38.862Z|00043|bridge|INFO|bridge br0: using datapath ID 0000fe022a4b4147
2017-12-14T15:32:38.862Z|00044|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-14T15:32:42.872Z|00045|memory|INFO|peak resident set size grew 54% in last 10.1 seconds, from 59268 kB to 91212 kB
2017-12-14T15:32:42.872Z|00046|memory|INFO|handlers:1 ports:1 revalidators:1 rules:5
2017-12-14T15:32:47.126Z|00047|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:32:47.514Z|00048|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:32:47.514Z|00049|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:32:55.867Z|00050|dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T15:32:56.266Z|00051|dpdk|ERR|Interface dpdk1 start error: Unknown error -4
2017-12-14T15:32:56.266Z|00052|dpif|WARN|netdev@ovs-netdev: failed to add dpdk1 as port: Unknown error -4
2017-12-14T15:32:56.607Z|00053|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:32:57.014Z|00054|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:32:57.015Z|00055|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:32:57.105Z|00056|timeval|WARN|Unreasonably long 1492ms poll interval (1297ms user, 0ms system)
2017-12-14T15:32:57.105Z|00057|timeval|WARN|faults: 106 minor, 0 major
2017-12-14T15:32:57.105Z|00058|timeval|WARN|context switches: 0 voluntary, 80 involuntary
2017-12-14T15:32:57.105Z|00059|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=2764ad03:
2017-12-14T15:32:57.105Z|00060|coverage|INFO|bridge_reconfigure         0.0/sec     0.067/sec        0.0011/sec   total: 5
2017-12-14T15:32:57.105Z|00061|coverage|INFO|ofproto_flush              0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00062|coverage|INFO|ofproto_update_port        0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:32:57.105Z|00063|coverage|INFO|rev_reconfigure            0.0/sec     0.050/sec        0.0008/sec   total: 4
2017-12-14T15:32:57.105Z|00064|coverage|INFO|rev_flow_table             0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00065|coverage|INFO|cmap_expand                0.0/sec     0.567/sec        0.0094/sec   total: 34
2017-12-14T15:32:57.105Z|00066|coverage|INFO|cmap_shrink                0.0/sec     0.317/sec        0.0053/sec   total: 19
2017-12-14T15:32:57.105Z|00067|coverage|INFO|dpif_port_add              0.0/sec     0.033/sec        0.0006/sec   total: 4
2017-12-14T15:32:57.105Z|00068|coverage|INFO|dpif_flow_flush            0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:32:57.105Z|00069|coverage|INFO|dpif_flow_get              0.0/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:32:57.105Z|00070|coverage|INFO|dpif_flow_put              0.0/sec     0.167/sec        0.0028/sec   total: 10
2017-12-14T15:32:57.105Z|00071|coverage|INFO|dpif_flow_del              0.0/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:32:57.105Z|00072|coverage|INFO|dpif_execute               0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:32:57.105Z|00073|coverage|INFO|flow_extract               0.0/sec     0.067/sec        0.0011/sec   total: 4
2017-12-14T15:32:57.105Z|00074|coverage|INFO|miniflow_malloc            0.0/sec     0.417/sec        0.0069/sec   total: 25
2017-12-14T15:32:57.105Z|00075|coverage|INFO|hmap_pathological          0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00076|coverage|INFO|hmap_expand               13.4/sec    16.817/sec        0.2803/sec   total: 1068
2017-12-14T15:32:57.105Z|00077|coverage|INFO|netdev_get_stats           0.2/sec     0.067/sec        0.0011/sec   total: 5
2017-12-14T15:32:57.105Z|00078|coverage|INFO|txn_unchanged              0.2/sec     0.100/sec        0.0017/sec   total: 8
2017-12-14T15:32:57.105Z|00079|coverage|INFO|txn_incomplete             0.0/sec     0.133/sec        0.0022/sec   total: 9
2017-12-14T15:32:57.105Z|00080|coverage|INFO|txn_success                0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:32:57.105Z|00081|coverage|INFO|poll_create_node          65.2/sec    19.400/sec        0.3233/sec   total: 1318
2017-12-14T15:32:57.105Z|00082|coverage|INFO|poll_zero_timeout          0.0/sec     0.067/sec        0.0011/sec   total: 4
2017-12-14T15:32:57.105Z|00083|coverage|INFO|seq_change               1038.4/sec   287.750/sec        4.7958/sec   total: 20388
2017-12-14T15:32:57.105Z|00084|coverage|INFO|pstream_open               0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:32:57.105Z|00085|coverage|INFO|stream_open                0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00086|coverage|INFO|util_xalloc              122.0/sec   282.783/sec        4.7131/sec   total: 17766
2017-12-14T15:32:57.105Z|00087|coverage|INFO|netdev_set_policing        0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:32:57.105Z|00088|coverage|INFO|netdev_get_ifindex         0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00089|coverage|INFO|netdev_get_hwaddr          0.0/sec     0.083/sec        0.0014/sec   total: 5
2017-12-14T15:32:57.105Z|00090|coverage|INFO|netdev_set_hwaddr          0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:32:57.105Z|00091|coverage|INFO|netdev_get_ethtool         0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:32:57.105Z|00092|coverage|INFO|netlink_received           0.2/sec     0.483/sec        0.0081/sec   total: 30
2017-12-14T15:32:57.105Z|00093|coverage|INFO|netlink_recv_jumbo         0.2/sec     0.067/sec        0.0011/sec   total: 5
2017-12-14T15:32:57.105Z|00094|coverage|INFO|netlink_sent               0.2/sec     0.267/sec        0.0044/sec   total: 17
2017-12-14T15:32:57.105Z|00095|coverage|INFO|nln_changed                0.0/sec     0.083/sec        0.0014/sec   total: 5
2017-12-14T15:32:57.105Z|00096|coverage|INFO|67 events never hit
2017-12-14T15:33:05.366Z|00097|dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T15:33:05.765Z|00098|dpdk|ERR|Interface dpdk1 start error: Unknown error -4
2017-12-14T15:33:05.766Z|00099|dpif|WARN|netdev@ovs-netdev: failed to add dpdk1 as port: Unknown error -4
2017-12-14T15:33:06.106Z|00100|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:33:06.515Z|00101|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:33:06.515Z|00102|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:33:06.604Z|00103|timeval|WARN|Unreasonably long 1489ms poll interval (1284ms user, 0ms system)
2017-12-14T15:33:06.604Z|00104|timeval|WARN|faults: 9 minor, 0 major
2017-12-14T15:33:06.604Z|00105|timeval|WARN|context switches: 0 voluntary, 73 involuntary
2017-12-14T15:33:06.605Z|00106|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=3f79bb11:
2017-12-14T15:33:06.605Z|00107|coverage|INFO|bridge_reconfigure         0.0/sec     0.083/sec        0.0014/sec   total: 6
2017-12-14T15:33:06.605Z|00108|coverage|INFO|ofproto_flush              0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00109|coverage|INFO|ofproto_update_port        0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:33:06.605Z|00110|coverage|INFO|rev_reconfigure            0.0/sec     0.067/sec        0.0011/sec   total: 5
2017-12-14T15:33:06.605Z|00111|coverage|INFO|rev_flow_table             0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00112|coverage|INFO|cmap_expand                0.0/sec     0.533/sec        0.0094/sec   total: 34
2017-12-14T15:33:06.605Z|00113|coverage|INFO|cmap_shrink                0.0/sec     0.317/sec        0.0053/sec   total: 19
2017-12-14T15:33:06.605Z|00114|coverage|INFO|dpif_port_add              0.0/sec     0.067/sec        0.0011/sec   total: 6
2017-12-14T15:33:06.605Z|00115|coverage|INFO|dpif_flow_flush            0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:33:06.605Z|00116|coverage|INFO|dpif_flow_get              0.0/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:33:06.605Z|00117|coverage|INFO|dpif_flow_put              0.0/sec     0.167/sec        0.0028/sec   total: 10
2017-12-14T15:33:06.605Z|00118|coverage|INFO|dpif_flow_del              0.0/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T15:33:06.605Z|00119|coverage|INFO|dpif_execute               0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:33:06.605Z|00120|coverage|INFO|flow_extract               0.0/sec     0.067/sec        0.0011/sec   total: 4
2017-12-14T15:33:06.605Z|00121|coverage|INFO|miniflow_malloc            0.0/sec     0.383/sec        0.0069/sec   total: 25
2017-12-14T15:33:06.605Z|00122|coverage|INFO|hmap_pathological          0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00123|coverage|INFO|hmap_expand               13.4/sec    16.583/sec        0.3214/sec   total: 1219
2017-12-14T15:33:06.605Z|00124|coverage|INFO|netdev_get_stats           0.2/sec     0.100/sec        0.0017/sec   total: 7
2017-12-14T15:33:06.605Z|00125|coverage|INFO|txn_unchanged              0.2/sec     0.150/sec        0.0025/sec   total: 11
2017-12-14T15:33:06.605Z|00126|coverage|INFO|txn_incomplete             0.0/sec     0.150/sec        0.0025/sec   total: 10
2017-12-14T15:33:06.605Z|00127|coverage|INFO|txn_success                0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:33:06.605Z|00128|coverage|INFO|poll_create_node          65.2/sec    28.867/sec        0.4839/sec   total: 1914
2017-12-14T15:33:06.605Z|00129|coverage|INFO|poll_zero_timeout          0.0/sec     0.067/sec        0.0011/sec   total: 4
2017-12-14T15:33:06.605Z|00130|coverage|INFO|seq_change               1038.4/sec   469.350/sec        7.8253/sec   total: 30262
2017-12-14T15:33:06.605Z|00131|coverage|INFO|pstream_open               0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:33:06.605Z|00132|coverage|INFO|stream_open                0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00133|coverage|INFO|util_xalloc              122.0/sec   266.167/sec        5.2161/sec   total: 19521
2017-12-14T15:33:06.605Z|00134|coverage|INFO|netdev_set_policing        0.0/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T15:33:06.605Z|00135|coverage|INFO|netdev_get_ifindex         0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00136|coverage|INFO|netdev_get_hwaddr          0.0/sec     0.083/sec        0.0014/sec   total: 5
2017-12-14T15:33:06.605Z|00137|coverage|INFO|netdev_set_hwaddr          0.0/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T15:33:06.605Z|00138|coverage|INFO|netdev_get_ethtool         0.0/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T15:33:06.605Z|00139|coverage|INFO|netlink_received           0.2/sec     0.517/sec        0.0086/sec   total: 32
2017-12-14T15:33:06.605Z|00140|coverage|INFO|netlink_recv_jumbo         0.2/sec     0.100/sec        0.0017/sec   total: 7
2017-12-14T15:33:06.605Z|00141|coverage|INFO|netlink_sent               0.2/sec     0.300/sec        0.0050/sec   total: 19
2017-12-14T15:33:06.605Z|00142|coverage|INFO|nln_changed                0.0/sec     0.083/sec        0.0014/sec   total: 5
2017-12-14T15:33:06.605Z|00143|coverage|INFO|67 events never hit
2017-12-14T15:33:11.870Z|00144|dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T15:33:12.265Z|00145|dpdk|ERR|Interface dpdk1 start error: Unknown error -4
2017-12-14T15:33:12.265Z|00146|dpif|WARN|netdev@ovs-netdev: failed to add dpdk1 as port: Unknown error -4
2017-12-14T15:33:12.606Z|00147|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:33:13.014Z|00148|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:33:13.014Z|00149|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:33:13.104Z|00150|timeval|WARN|Unreasonably long 1484ms poll interval (1279ms user, 0ms system)
2017-12-14T15:33:13.104Z|00151|timeval|WARN|faults: 1 minor, 0 major
2017-12-14T15:33:13.104Z|00152|timeval|WARN|context switches: 0 voluntary, 72 involuntary
2017-12-14T15:33:18.369Z|00153|dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T15:33:18.765Z|00154|dpdk|ERR|Interface dpdk1 start error: Unknown error -4
2017-12-14T15:33:18.765Z|00155|dpif|WARN|netdev@ovs-netdev: failed to add dpdk1 as port: Unknown error -4
2017-12-14T15:33:19.105Z|00156|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:33:19.515Z|00157|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:33:19.515Z|00158|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:33:19.604Z|00159|timeval|WARN|Unreasonably long 1485ms poll interval (1280ms user, 0ms system)
2017-12-14T15:33:19.604Z|00160|timeval|WARN|context switches: 0 voluntary, 70 involuntary
2017-12-14T15:33:19.861Z|00161|dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T15:33:20.213Z|00162|dpdk|ERR|Interface dpdk1 start error: Unknown error -4
2017-12-14T15:33:20.213Z|00163|dpif|WARN|netdev@ovs-netdev: failed to add dpdk1 as port: Unknown error -4
2017-12-14T15:33:20.556Z|00164|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T15:33:20.901Z|00165|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-14T15:33:20.901Z|00166|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
2017-12-14T15:33:20.990Z|00167|timeval|WARN|Unreasonably long 1383ms poll interval (1379ms user, 0ms system)
2017-12-14T15:33:20.990Z|00168|timeval|WARN|context switches: 0 voluntary, 72 involuntary
2017-12-14T15:33:20.994Z|00169|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 deletes)
2017-12-14T15:33:26.006Z|00170|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-12-14T15:33:26.008Z|00171|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)

Comment 66 Dilip Daya 2017-12-14 15:42:12 UTC
(In reply to Somnath Kotur from comment #63)
...
> 
> Hi Dilip,
>           Would have been nice if you had followed my steps as mentioned in
> comment# 48 as is just so we are on the same page.. 
> Having said that from what i see in your log, there is definite progress in
> the sense that OVS also is now detecting the bnxt ports (pasted below) which
> i don't believe you'd seen before ?

Correct, we have made progress and I had not seen these below messages before.


> > Starting ovs-vswitchd EAL: Detected 32 lcore(s)
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:12:00.0 on NUMA socket 0
> > EAL:   probe driver: 14e4:16d7 net_bnxt
> > EAL:   using IOMMU type 1 (Type 1)
> > PMD: Broadcom Cumulus driver bnxt
> > PMD: 1.8.0:20.6.135
> > PMD: Driver HWRM version: 1.5.1
> > PMD: BNXT Driver/HWRM API mismatch.
> > PMD: Firmware API version is newer than driver.
> > PMD: The driver may be missing features.
> > PMD: bnxt found at mem da600000, node addr 0x7f6c40000000M
> > EAL: PCI device 0000:12:00.1 on NUMA socket 0
> > EAL:   probe driver: 14e4:16d7 net_bnxt
> > PMD: 1.8.0:20.6.135
> > PMD: Driver HWRM version: 1.5.1
> > PMD: BNXT Driver/HWRM API mismatch.
> > PMD: Firmware API version is newer than driver.
> > PMD: The driver may be missing features.
> > PMD: bnxt found at mem da610000, node addr 0x7f6c40112000M
> > EAL: PCI device 0000:5d:00.0 on NUMA socket 0
> > EAL:   probe driver: 14e4:16d7 net_bnxt
> > EAL: PCI device 0000:5d:00.1 on NUMA socket 0
> > EAL:   probe driver: 14e4:16d7 net_bnxt
> > Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100,
> > virt:0x7f6c3ffced40, socket_id:0, flags:0
> >                                                            [  OK  ]
> > Enabling remote OVSDB managers                             [  OK  ]
> 
> Now, the only diff i can see between your steps and mine is that it appears
> as the way you have started your configuration database - the ovsdb-server
> 11)ovsdb-tool create /usr/local/etc/openvswitch/conf.db
> oswitchd/vswitch.ovsschema
> 12) rm -rf /usr/local/var/run/openvswitch
> 13) mkdir -p /usr/local/var/run/openvswitch
> 14) ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
> --private-key=db:Open_vSwitch,SSL,private_key 
> --certificate=db:Open_vSwitch,SSL,certificate
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach
> --log-file
> 15) export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> 
> and the subsequent initialization of this database like so:
> 16) ovs-vsctl --no-wait init
> 
> The only step i do for DPDK is this (below) before starting off the
> ovs-vswitchd
> 17) ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
> 
> Although like i said, the OVS has this time successfully detected the BNXT
> PMD and the error that you see seems OVS specific now


I went ahead and tried your steps in comment #48 earlier and was not successful. I've duplicated your comment #48 steps below:

[root@en104 openvswitch-2.6.1]# pwd
/root/rpmbuild/BUILD/openvswitch-2.6.1

[root@en104 openvswitch-2.6.1]# l /root/rpmbuild/BUILD/openvswitch-2.6.1/vswitchd/vswitch.ovsschema
-rw-r--r-- 1 paulc paulc 22966 Sep 28  2016 /root/rpmbuild/BUILD/openvswitch-2.6.1/vswitchd/vswitch.ovsschema

[root@en104 openvswitch-2.6.1]# find . -name vswitch.ovsschema
./vswitchd/vswitch.ovsschema


> 1) rpm -ivh <ovs.rpm> ( you can use the same  ovs-2.6.1.src.rpm that you have/attached to this bug)
> 2) cd /root/rpmbuild/SPECS
> 3) rpmbuild -bp openvswitch.spec(NO change) (you might need to 'yum install' some other pkgs to get this working)
> 4) cd ../BUILD/openvswitch-2.6.1
> 5) ./configure --with-dpdk=$DPDK_BUILD (set DPDK_BUILD and other DPDK_* env variables to the extracted folder from the tarball that i provided)
> 6) make (Type `autoreconf -f -i` if it complains about aclocal-1.14 command 
         not found on your system)
> 7) make install
> 8) make modules_install
> 9) modprobe openvswitch
> 10) export PATH=$PATH:/usr/local/share/openvswitch/scripts

> Now, the only diff i can see between your steps and mine is that it appears as the way you have started your configuration database - the ovsdb-server
> 11) ovsdb-tool create /usr/local/etc/openvswitch/conf.db oswitchd/vswitch.ovsschema

mkdir -p /usr/local/etc/openvswitch

[root@en104 openvswitch-2.6.1]# find . -name vswitch.ovsschema
./vswitchd/vswitch.ovsschema

[root@en104 openvswitch-2.6.1]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db ./vswitchd/vswitch.ovsschema
[root@en104 openvswitch-2.6.1]# l /usr/local/etc/openvswitch
total 16
-rw-r--r-- 1 root root 13038 Dec 14 07:49 conf.db


> 12) rm -rf /usr/local/var/run/openvswitch

[root@en104 openvswitch-2.6.1]# rm -rf /usr/local/var/run/openvswitch


> 13) mkdir -p /usr/local/var/run/openvswitch

[root@en104 openvswitch-2.6.1]# mkdir -p /usr/local/var/run/openvswitch

> 14) ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file

[root@en104 openvswitch-2.6.1]# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key  --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert  --pidfile --detach --log-file
2017-12-14T14:51:34Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovsdb-server.log


> 15) export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

[root@en104 openvswitch-2.6.1]# l /usr/local/var/run/openvswitch/db.sock
srwxr-x--- 1 root root 0 Dec 14 07:51 /usr/local/var/run/openvswitch/db.sock=

[root@en104 openvswitch-2.6.1]# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock


> and the subsequent initialization of this database like so:
> 16) ovs-vsctl --no-wait init

[root@en104 openvswitch-2.6.1]# ovs-vsctl --no-wait init
[root@en104 openvswitch-2.6.1]# 


> The only step i do for DPDK is this (below) before starting off the ovs-vswitchd
> 17) ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

[root@en104 openvswitch-2.6.1]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
[root@en104 openvswitch-2.6.1]# 


> 18) ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start

[root@en104 openvswitch-2.6.1]# find . -name ovs-ctl
./utilities/ovs-ctl
[root@en104 openvswitch-2.6.1]# l ./utilities/ovs-ctl
-rwxr-xr-x 1 root root 21866 Dec 13 16:56 ./utilities/ovs-ctl*

[root@en104 openvswitch-2.6.1]# find /usr/local/share/openvswitch/scripts -name ovs-ctl
/usr/local/share/openvswitch/scripts/ovs-ctl
[root@en104 openvswitch-2.6.1]# l /usr/local/share/openvswitch/scripts/ovs-ctl
-rwxr-xr-x 1 root root 21866 Dec 13 12:34 /usr/local/share/openvswitch/scripts/ovs-ctl*

[root@en104 openvswitch-2.6.1]# env | grep DB_SOCK
DB_SOCK=/usr/local/var/run/openvswitch/db.sock
[root@en104 openvswitch-2.6.1]# l /usr/local/var/run/openvswitch/db.sock
srwxr-x--- 1 root root 0 Dec 14 07:51 /usr/local/var/run/openvswitch/db.sock=

[root@en104 openvswitch-2.6.1]# ./utilities/ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
Inserting openvswitch module                               [  OK  ]
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: Master core has no memory on local socket!
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PANIC in rte_eth_dev_data_alloc():
Cannot allocate memzone for ethernet port data  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
13: [ovs-vswitchd() [0x40fc03]]
12: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f635ca76c05]]
11: [ovs-vswitchd() [0x40eea5]]
10: [ovs-vswitchd() [0x5c4c3a]]
9: [ovs-vswitchd() [0x6c6bda]]
8: [ovs-vswitchd() [0x6c66c0]]
7: [ovs-vswitchd() [0x561d4f]]
6: [ovs-vswitchd() [0x56cfe2]]
5: [ovs-vswitchd() [0x56caea]]
4: [ovs-vswitchd() [0x4408fd]]
3: [ovs-vswitchd() [0x44088a]]
2: [ovs-vswitchd() [0x40b928]]
1: [ovs-vswitchd() [0x56911a]]
ovs-vswitchd: could not initiate process monitoring
                                                           [FAILED]
[root@en104 openvswitch-2.6.1]# 


[root@en104 openvswitch-2.6.1]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 97964 MB
node 0 free: 25955 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 1 size: 98303 MB
node 1 free: 23314 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

=> And that is why I used my scripts to start OVS as documented in comment #62.


> 2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device
> dpdkport0 (No such device)
> 
> A quick search yielded a known RH bug and a workaround for this symptom 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1397299
> 
> You might want to try this as well ?

BZ #1397299 relates to RH startup scripts and in our case we're using commandline only as in comment #62.

> 
> Ajit,
>       Could you also please set this up on your systems so you could guide
> Dilip runtime in his timezone ?
> 
> Thanks
> Som


Thanks,
-DilipD.

Comment 67 Dilip Daya 2017-12-14 18:34:18 UTC
Would this help in diagnosing this issue:

++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7f9ac0000000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7f9ac0112000M
EAL: PCI device 0000:5d:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL: PCI device 0000:5d:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100, virt:0x7f9abffced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]


=> compared to working upstream ovs:

# Start the ovs-vswitchd:
++ /home/dilip/openvswitch-2.8.1/utilities/ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd Zone 0: name:<rte_eth_dev_data>, phys:0x83ffb6dc0, len:0x34000, virt:0x7f317ffb6dc0, socket_id:0, flags:0
Zone 1: name:<bnxt_0000:12:00:00-rx_port_stat>, phys:0x83ffb6300, len:0x440, virt:0x7f317ffb6300, socket_id:0, flags:0
Zone 2: name:<bnxt_0000:12:00:00-tx_port_stat>, phys:0x83ffb5f00, len:0x3c0, virt:0x7f317ffb5f00, socket_id:0, flags:0
Zone 3: name:<bnxt_0000:12:00:00-0000_def_cp>, phys:0x83ffb4000, len:0x1000, virt:0x7f317ffb4000, socket_id:0, flags:0
Zone 4: name:<bnxt_0000:12:00:00_vnicattr>, phys:0x83ff8e1c0, len:0x1be40, virt:0x7f317ff8e1c0, socket_id:0, flags:0
Zone 5: name:<bnxt_0000:12:00:01-rx_port_stat>, phys:0x83ff82fc0, len:0x440, virt:0x7f317ff82fc0, socket_id:0, flags:0
Zone 6: name:<bnxt_0000:12:00:01-tx_port_stat>, phys:0x83ff82bc0, len:0x3c0, virt:0x7f317ff82bc0, socket_id:0, flags:0
Zone 7: name:<bnxt_0000:12:00:01-0000_def_cp>, phys:0x83ff81000, len:0x1000, virt:0x7f317ff81000, socket_id:0, flags:0
Zone 8: name:<bnxt_0000:12:00:01_vnicattr>, phys:0x83ff5af80, len:0x1c000, virt:0x7f317ff5af80, socket_id:0, flags:0
                                  [  OK  ]
Enabling remote OVSDB managers    [  OK  ]
...

Comment 68 Dilip Daya 2017-12-15 01:18:19 UTC
Environment
-----------

* RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
* openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
* Modified dpdk tree (dpdk-stable-16.11.1) within openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
  - Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in config/common_linuxapp
  - Applied patch from:
    - http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415



+ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/dpdk-stable-16.11.1/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


[root@en104 rhel-7-fast-datapath_packages]# ./start_ovs_and_looped_flows_no_comments_git20170530.el7fdb.sh 
++ export OVSSCHEMA=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/vswitchd/vswitch.ovsschema
++ OVSSCHEMA=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/vswitchd/vswitch.ovsschema
++ export OVS_DIR=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0
++ OVS_DIR=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0
++ export OVS_UTILS=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities
++ OVS_UTILS=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities
++ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ DB_SOCK=/usr/local/var/run/openvswitch/db.sock
++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
++ P1=0000:12:00.0
++ P2=0000:12:00.1
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ctl stop
ovs-vswitchd is not running.
ovsdb-server is not running.
++ pkill -9 ovs
++ rm -rf /usr/local/var/run/openvswitch
++ rm -rf /usr/local/etc/openvswitch/
++ rm -rf /usr/local/etc/openvswitch/conf.db
++ rm -fr /usr/local/var/log/openvswitch/
++ mkdir -p /usr/local/etc/openvswitch
++ mkdir -p /usr/local/var/run/openvswitch
++ mkdir -p /usr/local/var/log/openvswitch
++ modprobe openvswitch
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/vswitchd/vswitch.ovsschema
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ctl --system-id=random --no-ovs-vswitchd start
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=1
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=2048,0
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6
++ sleep 6[root@en104 openvswitch-2.7.0]# ./utilities/ovs-vsctl show
9f085008-4b7d-401d-b7f9-1a4399324c4c
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "dpdkport1"
            Interface "dpdkport1"
                type: dpdk
                options: {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
        Port "dpdkport0"
            Interface "dpdkport0"
                type: dpdk
                options: {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
    ovs_version: "2.7.0"

++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=30000
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ctl --no-ovsdb-server --db-sock=/usr/local/var/run/openvswitch/db.sock start
Starting ovs-vswitchd EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:12:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da600000, node addr 0x7fcf80000000M
EAL: PCI device 0000:12:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.5.1
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem da610000, node addr 0x7fcf80112000M
EAL: PCI device 0000:5d:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL: PCI device 0000:5d:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100, virt:0x7fcf7ffced40, socket_id:0, flags:0
                                                           [  OK  ]
Enabling remote OVSDB managers                             [  OK  ]
++ sleep 6
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ctl status
ovsdb-server is running with pid 12422
ovs-vswitchd is running with pid 12503
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
++ sleep 8
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl add-port br0 dpdkport0 -- set Interface dpdkport0 type=dpdk options:dpdk-devargs=0000:12:00.0 ofport_request=1
++ sleep 8
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl add-port br0 dpdkport1 -- set Interface dpdkport1 type=dpdk options:dpdk-devargs=0000:12:00.1 ofport_request=2
++ sleep 8
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport0 options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
++ sleep 5
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport1 options:n_rxq=1 other_config:pmd-rxq-affinity=0:2
++ sleep 5
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport0 options:n_rxq_desc=2048
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport1 options:n_rxq_desc=2048
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=34.603s, table=0, n_packets=0, n_bytes=0, idle_age=34, priority=0 actions=NORMAL
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl del-flows br0
++ sleep 5
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl add-flow br0 in_port=1,action=output:2
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl add-flow br0 in_port=2,action=output:1
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, in_port=2 actions=output:1
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:00009cdc71b3b8a0
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(dpdkport0): addr:9c:dc:71:b3:b8:a0
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(dpdkport1): addr:9c:dc:71:b3:b8:a8
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br0): addr:9c:dc:71:b3:b8:a0
     config:     PORT_DOWN
     state:      LINK_DOWN
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-appctl dpif-netdev/pmd-stats-show
main thread:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:734798 (100.00%)
	processing cycles:0 (0.00%)
pmd thread numa_id 0 core_id 1:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:28979504064 (100.00%)
	processing cycles:0 (0.00%)
pmd thread numa_id 0 core_id 2:
	emc hits:0
	megaflow hits:0
	avg. subtable lookups per hit:0.00
	miss:0
	lost:0
	polling cycles:21378576222 (100.00%)
	processing cycles:0 (0.00%)
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 1:
	isolated : true
	port: dpdkport0	queue-id: 0
pmd thread numa_id 0 core_id 2:
	isolated : true
	port: dpdkport1	queue-id: 0
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 3 ports
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
  port  1: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=3, bytes=258, drop=0, errs=0, coll=?
  port  2: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=6, bytes=500, drop=0, errs=0, coll=?
++ /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/openvswitch-2.7.0/utilities/ovs-vsctl list interface
_uuid               : 6bc29ad1-ad0d-48d4-a229-e0d75f62e0ca
admin_state         : down
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
error               : []
external_ids        : {}
ifindex             : 15
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : 10000000
link_state          : down
lldp                : {}
mac                 : []
mac_in_use          : "9c:dc:71:b3:b8:a0"
mtu                 : 1500
mtu_request         : []
name                : "br0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status              : {driver_name=tun, driver_version="1.6", firmware_version=""}
type                : internal

_uuid               : a77ea73e-f2a8-4672-9ca7-4ff24c088cbf
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : []
external_ids        : {}
ifindex             : 1
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : []
link_state          : up
lldp                : {}
mac                 : []
mac_in_use          : "9c:dc:71:b3:b8:a8"
mtu                 : 1500
mtu_request         : []
name                : "dpdkport1"
ofport              : 2
ofport_request      : 2
options             : {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:2"}
statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0, tx_bytes=500, tx_dropped=0, tx_errors=0, tx_packets=6}
status              : {driver_name=net_bnxt, max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="241", max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1", numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="1"}
type                : dpdk

_uuid               : 3dc3d88b-9a62-43fc-a467-efa376bb5fa8
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
error               : []
external_ids        : {}
ifindex             : 0
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 0
link_speed          : []
link_state          : up
lldp                : {}
mac                 : []
mac_in_use          : "9c:dc:71:b3:b8:a0"
mtu                 : 1500
mtu_request         : []
name                : "dpdkport0"
ofport              : 1
ofport_request      : 1
options             : {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
other_config        : {pmd-rxq-affinity="0:1"}
statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0, tx_bytes=258, tx_dropped=0, tx_errors=0, tx_packets=3}
status              : {driver_name=net_bnxt, max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="242", max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1", numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="0"}
type                : dpdk
++ ps -ef
++ grep ovs
root     12421     1  0 16:35 ?        00:00:00 ovsdb-server: monitoring pid 12422 (healthy)
root     12422 12421  0 16:35 ?        00:00:00 ovsdb-server /usr/local/etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/usr/local/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root     12502     1  0 16:36 ?        00:00:00 ovs-vswitchd: monitoring pid 12503 (healthy)
root     12503 12502 99 16:36 ?        00:01:35 ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
root     12647 12368  0 16:37 pts/7    00:00:00 grep ovs
[root@en104 rhel-7-fast-datapath_packages]# 
[root@en104 rhel-7-fast-datapath_packages]# 
[root@en104 rhel-7-fast-datapath_packages]# 
[root@en104 rhel-7-fast-datapath_packages]# cat /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-14T23:36:28.010Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-14T23:36:28.014Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0
2017-12-14T23:36:28.014Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1
2017-12-14T23:36:28.014Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores
2017-12-14T23:36:28.014Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
2017-12-14T23:36:28.014Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
2017-12-14T23:36:28.015Z|00007|dpdk|INFO|DPDK Enabled - initializing...
2017-12-14T23:36:28.015Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /usr/local/var/run/openvswitch
2017-12-14T23:36:28.015Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1 --socket-mem 2048,0
2017-12-14T23:37:08.707Z|00010|dpdk|INFO|DPDK Enabled - initialized
2017-12-14T23:37:08.708Z|00011|timeval|WARN|Unreasonably long 40692ms poll interval (40ms user, 40050ms system)
2017-12-14T23:37:08.708Z|00012|timeval|WARN|faults: 81100 minor, 0 major
2017-12-14T23:37:08.708Z|00013|timeval|WARN|disk: 0 reads, 8 writes
2017-12-14T23:37:08.708Z|00014|timeval|WARN|context switches: 5 voluntary, 48 involuntary
2017-12-14T23:37:08.708Z|00015|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=fde89638:
2017-12-14T23:37:08.708Z|00016|coverage|INFO|bridge_reconfigure         0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00017|coverage|INFO|cmap_expand                0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T23:37:08.708Z|00018|coverage|INFO|miniflow_malloc            0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-14T23:37:08.708Z|00019|coverage|INFO|hmap_pathological          0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00020|coverage|INFO|hmap_expand               10.2/sec     6.333/sec        0.1056/sec   total: 380
2017-12-14T23:37:08.708Z|00021|coverage|INFO|txn_unchanged              0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T23:37:08.708Z|00022|coverage|INFO|txn_incomplete             0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00023|coverage|INFO|poll_create_node           1.0/sec     0.667/sec        0.0111/sec   total: 40
2017-12-14T23:37:08.708Z|00024|coverage|INFO|seq_change                 1.8/sec     0.733/sec        0.0122/sec   total: 44
2017-12-14T23:37:08.708Z|00025|coverage|INFO|pstream_open               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00026|coverage|INFO|stream_open                0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00027|coverage|INFO|util_xalloc              182.6/sec   121.383/sec        2.0231/sec   total: 7283
2017-12-14T23:37:08.708Z|00028|coverage|INFO|netdev_get_hwaddr          0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-14T23:37:08.708Z|00029|coverage|INFO|netlink_received           0.6/sec     0.050/sec        0.0008/sec   total: 3
2017-12-14T23:37:08.708Z|00030|coverage|INFO|netlink_sent               0 errs=5703523.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-14T23:37:08.708Z|00031|coverage|INFO|86 events never hit
2017-12-14T23:37:08.710Z|00032|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98% CPU usage)
2017-12-14T23:37:08.710Z|00033|memory|INFO|59108 kB peak resident set size after 40.7 seconds
2017-12-14T23:37:08.710Z|00034|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.7.0
2017-12-14T23:37:14.753Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-12-14T23:37:14.753Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1
2017-12-14T23:37:14.753Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-12-14T23:37:14.753Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action
2017-12-14T23:37:14.753Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-12-14T23:37:14.753Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state
2017-12-14T23:37:14.753Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone
2017-12-14T23:37:14.753Z|00042|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark
2017-12-14T23:37:14.753Z|00043|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label
2017-12-14T23:37:14.753Z|00044|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state_nat
2017-12-14T23:37:14.762Z|00045|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-14T23:37:14.762Z|00046|bridge|INFO|bridge br0: using datapath ID 0000ca98fcc4fa43
2017-12-14T23:37:14.762Z|00047|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-14T23:37:22.782Z|00048|netdev_dpdk|WARN|Rx checksum offload is not supported on device 0
2017-12-14T23:37:22.788Z|00049|dpif_netdev|INFO|Created 2 pmd threads on numa node 0
2017-12-14T23:37:23.048Z|00050|netdev_dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-14T23:37:23.049Z|00051|bridge|INFO|bridge br0: added interface dpdkport0 on port 1
2017-12-14T23:37:23.049Z|00052|bridge|INFO|bridge br0: using datapath ID 00009cdc71b3b8a0
2017-12-14T23:37:29.228Z|00053|memory|INFO|peak resident set size grew 89% in last 20.5 seconds, from 59108 kB to 111796 kB
2017-12-14T23:37:29.229Z|00054|memory|INFO|handlers:1 ports:2 revalidators:1 rules:5
2017-12-14T23:37:31.209Z|00055|netdev_dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-14T23:37:31.209Z|00056|bridge|INFO|bridge br0: added interface dpdkport1 on port 2
2017-12-14T23:37:49.367Z|00057|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 deletes)
2017-12-14T23:37:54.376Z|00058|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-12-14T23:37:54.378Z|00059|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)


[root@en104 openvswitch-2.7.0]# ./utilities/ovs-vsctl show
9f085008-4b7d-401d-b7f9-1a4399324c4c
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "dpdkport1"
            Interface "dpdkport1"
                type: dpdk
                options: {dpdk-devargs="0000:12:00.1", n_rxq="1", n_rxq_desc="2048"}
        Port "dpdkport0"
            Interface "dpdkport0"
                type: dpdk
                options: {dpdk-devargs="0000:12:00.0", n_rxq="1", n_rxq_desc="2048"}
    ovs_version: "2.7.0"



=> Generated Spirent traffic:

# ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=3286.471s, table=0, n_packets=633017838, n_bytes=37981070280, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=3286.470s, table=0, n_packets=694627989, n_bytes=41677679340, idle_age=0, in_port=2 actions=output:1

Comment 69 Somnath Kotur 2017-12-15 06:54:50 UTC
(In reply to Dilip Daya from comment #68)
> Environment
> -----------
> 
> * RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> * openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
> * Modified dpdk tree (dpdk-stable-16.11.1) within
> openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
>   - Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
> config/common_linuxapp
>   - Applied patch from:
>     -
> http://www.dpdk.org/browse/dpdk/commit/
> ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415
> 
> 
> 
> +
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/dpdk-stable-16.11.1/tools/dpdk-devbind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
> drv=vfio-pci unused=bnxt_en
> 
> 
> [root@en104 rhel-7-fast-datapath_packages]#
> ./start_ovs_and_looped_flows_no_comments_git20170530.el7fdb.sh 
> ++ export
> OVSSCHEMA=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0/vswitchd/vswitch.ovsschema
> ++
> OVSSCHEMA=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0/vswitchd/vswitch.ovsschema
> ++ export
> OVS_DIR=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0
> ++
> OVS_DIR=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0
> ++ export
> OVS_UTILS=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0/utilities
> ++
> OVS_UTILS=/root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/
> BUILD/openvswitch-2.7.0/utilities
> ++ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> ++ DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> ++ export
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/
> share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
> ++
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/
> share/openvswitch/scripts:/usr/local/share/openvswitch/scripts
> ++ P1=0000:12:00.0
> ++ P2=0000:12:00.1
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ctl stop
> ovs-vswitchd is not running.
> ovsdb-server is not running.
> ++ pkill -9 ovs
> ++ rm -rf /usr/local/var/run/openvswitch
> ++ rm -rf /usr/local/etc/openvswitch/
> ++ rm -rf /usr/local/etc/openvswitch/conf.db
> ++ rm -fr /usr/local/var/log/openvswitch/
> ++ mkdir -p /usr/local/etc/openvswitch
> ++ mkdir -p /usr/local/var/run/openvswitch
> ++ mkdir -p /usr/local/var/log/openvswitch
> ++ modprobe openvswitch
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/vswitchd/vswitch.ovsschema
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ctl --system-id=random --no-ovs-vswitchd
> start
> Starting ovsdb-server                                      [  OK  ]
> Configuring Open vSwitch system IDs                        [  OK  ]
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-lcore-mask=1
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-socket-mem=2048,0
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:pmd-cpu-mask=6
> ++ sleep 6[root@en104 openvswitch-2.7.0]# ./utilities/ovs-vsctl show
> 9f085008-4b7d-401d-b7f9-1a4399324c4c
>     Bridge "br0"
>         Port "br0"
>             Interface "br0"
>                 type: internal
>         Port "dpdkport1"
>             Interface "dpdkport1"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:12:00.1", n_rxq="1",
> n_rxq_desc="2048"}
>         Port "dpdkport0"
>             Interface "dpdkport0"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:12:00.0", n_rxq="1",
> n_rxq_desc="2048"}
>     ovs_version: "2.7.0"
> 
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-init=true
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl --no-wait set Open_vSwitch .
> other_config:max-idle=30000
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ctl --no-ovsdb-server
> --db-sock=/usr/local/var/run/openvswitch/db.sock start
> Starting ovs-vswitchd EAL: Detected 32 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:12:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL:   using IOMMU type 1 (Type 1)
> PMD: Broadcom Cumulus driver bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da600000, node addr 0x7fcf80000000M
> EAL: PCI device 0000:12:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> PMD: 1.8.0:20.6.135
> PMD: Driver HWRM version: 1.5.1
> PMD: BNXT Driver/HWRM API mismatch.
> PMD: Firmware API version is newer than driver.
> PMD: The driver may be missing features.
> PMD: bnxt found at mem da610000, node addr 0x7fcf80112000M
> EAL: PCI device 0000:5d:00.0 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> EAL: PCI device 0000:5d:00.1 on NUMA socket 0
> EAL:   probe driver: 14e4:16d7 net_bnxt
> Zone 0: name:<rte_eth_dev_data>, phys:0x83ffced40, len:0x30100,
> virt:0x7fcf7ffced40, socket_id:0, flags:0
>                                                            [  OK  ]
> Enabling remote OVSDB managers                             [  OK  ]
> ++ sleep 6
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ctl status
> ovsdb-server is running with pid 12422
> ovs-vswitchd is running with pid 12503
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl add-br br0 -- set bridge br0
> datapath_type=netdev
> ++ sleep 8
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl add-port br0 dpdkport0 -- set
> Interface dpdkport0 type=dpdk options:dpdk-devargs=0000:12:00.0
> ofport_request=1
> ++ sleep 8
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl add-port br0 dpdkport1 -- set
> Interface dpdkport1 type=dpdk options:dpdk-devargs=0000:12:00.1
> ofport_request=2
> ++ sleep 8
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport0
> options:n_rxq=1 other_config:pmd-rxq-affinity=0:1
> ++ sleep 5
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport1
> options:n_rxq=1 other_config:pmd-rxq-affinity=0:2
> ++ sleep 5
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport0
> options:n_rxq_desc=2048
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl set Interface dpdkport1
> options:n_rxq_desc=2048
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl dump-flows br0
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=34.603s, table=0, n_packets=0, n_bytes=0, idle_age=34,
> priority=0 actions=NORMAL
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl del-flows br0
> ++ sleep 5
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl add-flow br0 in_port=1,action=output:2
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl add-flow br0 in_port=2,action=output:1
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl dump-flows br0
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0,
> in_port=1 actions=output:2
>  cookie=0x0, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0,
> in_port=2 actions=output:1
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl show br0
> OFPT_FEATURES_REPLY (xid=0x2): dpid:00009cdc71b3b8a0
> n_tables:254, n_buffers:0
> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
> actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src
> mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
>  1(dpdkport0): addr:9c:dc:71:b3:b8:a0
>      config:     0
>      state:      0
>      speed: 0 Mbps now, 0 Mbps max
>  2(dpdkport1): addr:9c:dc:71:b3:b8:a8
>      config:     0
>      state:      0
>      speed: 0 Mbps now, 0 Mbps max
>  LOCAL(br0): addr:9c:dc:71:b3:b8:a0
>      config:     PORT_DOWN
>      state:      LINK_DOWN
>      current:    10MB-FD COPPER
>      speed: 10 Mbps now, 0 Mbps max
> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-appctl dpif-netdev/pmd-stats-show
> main thread:
> 	emc hits:0
> 	megaflow hits:0
> 	avg. subtable lookups per hit:0.00
> 	miss:0
> 	lost:0
> 	polling cycles:734798 (100.00%)
> 	processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 1:
> 	emc hits:0
> 	megaflow hits:0
> 	avg. subtable lookups per hit:0.00
> 	miss:0
> 	lost:0
> 	polling cycles:28979504064 (100.00%)
> 	processing cycles:0 (0.00%)
> pmd thread numa_id 0 core_id 2:
> 	emc hits:0
> 	megaflow hits:0
> 	avg. subtable lookups per hit:0.00
> 	miss:0
> 	lost:0
> 	polling cycles:21378576222 (100.00%)
> 	processing cycles:0 (0.00%)
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-appctl dpif-netdev/pmd-rxq-show
> pmd thread numa_id 0 core_id 1:
> 	isolated : true
> 	port: dpdkport0	queue-id: 0
> pmd thread numa_id 0 core_id 2:
> 	isolated : true
> 	port: dpdkport1	queue-id: 0
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-ofctl dump-ports br0
> OFPST_PORT reply (xid=0x2): 3 ports
>   port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
>            tx pkts=0, bytes=0, drop=0, errs=0, coll=0
>   port  1: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=?
>            tx pkts=3, bytes=258, drop=0, errs=0, coll=?
>   port  2: rx pkts=0, bytes=0, drop=0, errs=0, frame=?, over=?, crc=?
>            tx pkts=6, bytes=500, drop=0, errs=0, coll=?
> ++
> /root/rpmbuild_openvswitch-2.7.0-8.git20170530.el7fdb_working_dpdk/BUILD/
> openvswitch-2.7.0/utilities/ovs-vsctl list interface
> _uuid               : 6bc29ad1-ad0d-48d4-a229-e0d75f62e0ca
> admin_state         : down
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : full
> error               : []
> external_ids        : {}
> ifindex             : 15
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : 0
> link_speed          : 10000000
> link_state          : down
> lldp                : {}
> mac                 : []
> mac_in_use          : "9c:dc:71:b3:b8:a0"
> mtu                 : 1500
> mtu_request         : []
> name                : "br0"
> ofport              : 65534
> ofport_request      : []
> options             : {}
> other_config        : {}
> statistics          : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0,
> rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0,
> tx_dropped=0, tx_errors=0, tx_packets=0}
> status              : {driver_name=tun, driver_version="1.6",
> firmware_version=""}
> type                : internal
> 
> _uuid               : a77ea73e-f2a8-4672-9ca7-4ff24c088cbf
> admin_state         : up
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : []
> error               : []
> external_ids        : {}
> ifindex             : 1
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : 0
> link_speed          : []
> link_state          : up
> lldp                : {}
> mac                 : []
> mac_in_use          : "9c:dc:71:b3:b8:a8"
> mtu                 : 1500
> mtu_request         : []
> name                : "dpdkport1"
> ofport              : 2
> ofport_request      : 2
> options             : {dpdk-devargs="0000:12:00.1", n_rxq="1",
> n_rxq_desc="2048"}
> other_config        : {pmd-rxq-affinity="0:2"}
> statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0,
> tx_bytes=500, tx_dropped=0, tx_errors=0, tx_packets=6}
> status              : {driver_name=net_bnxt, max_hash_mac_addrs="0",
> max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="241",
> max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1",
> numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="1"}
> type                : dpdk
> 
> _uuid               : 3dc3d88b-9a62-43fc-a467-efa376bb5fa8
> admin_state         : up
> bfd                 : {}
> bfd_status          : {}
> cfm_fault           : []
> cfm_fault_status    : []
> cfm_flap_count      : []
> cfm_health          : []
> cfm_mpid            : []
> cfm_remote_mpids    : []
> cfm_remote_opstate  : []
> duplex              : []
> error               : []
> external_ids        : {}
> ifindex             : 0
> ingress_policing_burst: 0
> ingress_policing_rate: 0
> lacp_current        : []
> link_resets         : 0
> link_speed          : []
> link_state          : up
> lldp                : {}
> mac                 : []
> mac_in_use          : "9c:dc:71:b3:b8:a0"
> mtu                 : 1500
> mtu_request         : []
> name                : "dpdkport0"
> ofport              : 1
> ofport_request      : 1
> options             : {dpdk-devargs="0000:12:00.0", n_rxq="1",
> n_rxq_desc="2048"}
> other_config        : {pmd-rxq-affinity="0:1"}
> statistics          : {rx_bytes=0, rx_dropped=0, rx_errors=0, rx_packets=0,
> tx_bytes=258, tx_dropped=0, tx_errors=0, tx_packets=3}
> status              : {driver_name=net_bnxt, max_hash_mac_addrs="0",
> max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="242",
> max_tx_queues="170", max_vfs="0", max_vmdq_pools="64", min_rx_bufsize="1",
> numa_id="0", pci-device_id="0x16d7", pci-vendor_id="0x5348", port_no="0"}
> type                : dpdk
> ++ ps -ef
> ++ grep ovs
> root     12421     1  0 16:35 ?        00:00:00 ovsdb-server: monitoring pid
> 12422 (healthy)
> root     12422 12421  0 16:35 ?        00:00:00 ovsdb-server
> /usr/local/etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info
> --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --private-key=db:Open_vSwitch,SSL,private_key
> --certificate=db:Open_vSwitch,SSL,certificate
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir
> --log-file=/usr/local/var/log/openvswitch/ovsdb-server.log
> --pidfile=/usr/local/var/run/openvswitch/ovsdb-server.pid --detach --monitor
> root     12502     1  0 16:36 ?        00:00:00 ovs-vswitchd: monitoring pid
> 12503 (healthy)
> root     12503 12502 99 16:36 ?        00:01:35 ovs-vswitchd
> unix:/usr/local/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
> -vfile:info --mlockall --no-chdir
> --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/usr/local/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
> root     12647 12368  0 16:37 pts/7    00:00:00 grep ovs
> [root@en104 rhel-7-fast-datapath_packages]# 
> [root@en104 rhel-7-fast-datapath_packages]# 
> [root@en104 rhel-7-fast-datapath_packages]# 
> [root@en104 rhel-7-fast-datapath_packages]# cat
> /usr/local/var/log/openvswitch/ovs-vswitchd.log
> 2017-12-14T23:36:28.010Z|00001|vlog|INFO|opened log file
> /usr/local/var/log/openvswitch/ovs-vswitchd.log
> 2017-12-14T23:36:28.014Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA
> node 0
> 2017-12-14T23:36:28.014Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA
> node 1
> 2017-12-14T23:36:28.014Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32
> CPU cores
> 2017-12-14T23:36:28.014Z|00005|reconnect|INFO|unix:/usr/local/var/run/
> openvswitch/db.sock: connecting...
> 2017-12-14T23:36:28.014Z|00006|reconnect|INFO|unix:/usr/local/var/run/
> openvswitch/db.sock: connected
> 2017-12-14T23:36:28.015Z|00007|dpdk|INFO|DPDK Enabled - initializing...
> 2017-12-14T23:36:28.015Z|00008|dpdk|INFO|No vhost-sock-dir provided -
> defaulting to /usr/local/var/run/openvswitch
> 2017-12-14T23:36:28.015Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1
> --socket-mem 2048,0
> 2017-12-14T23:37:08.707Z|00010|dpdk|INFO|DPDK Enabled - initialized
> 2017-12-14T23:37:08.708Z|00011|timeval|WARN|Unreasonably long 40692ms poll
> interval (40ms user, 40050ms system)
> 2017-12-14T23:37:08.708Z|00012|timeval|WARN|faults: 81100 minor, 0 major
> 2017-12-14T23:37:08.708Z|00013|timeval|WARN|disk: 0 reads, 8 writes
> 2017-12-14T23:37:08.708Z|00014|timeval|WARN|context switches: 5 voluntary,
> 48 involuntary
> 2017-12-14T23:37:08.708Z|00015|coverage|INFO|Event coverage, avg rate over
> last: 5 seconds, last minute, last hour,  hash=fde89638:
> 2017-12-14T23:37:08.708Z|00016|coverage|INFO|bridge_reconfigure        
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00017|coverage|INFO|cmap_expand               
> 0.4/sec     0.150/sec        0.0025/sec   total: 9
> 2017-12-14T23:37:08.708Z|00018|coverage|INFO|miniflow_malloc           
> 0.4/sec     0.150/sec        0.0025/sec   total: 9
> 2017-12-14T23:37:08.708Z|00019|coverage|INFO|hmap_pathological         
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00020|coverage|INFO|hmap_expand              
> 10.2/sec     6.333/sec        0.1056/sec   total: 380
> 2017-12-14T23:37:08.708Z|00021|coverage|INFO|txn_unchanged             
> 0.4/sec     0.033/sec        0.0006/sec   total: 2
> 2017-12-14T23:37:08.708Z|00022|coverage|INFO|txn_incomplete            
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00023|coverage|INFO|poll_create_node          
> 1.0/sec     0.667/sec        0.0111/sec   total: 40
> 2017-12-14T23:37:08.708Z|00024|coverage|INFO|seq_change                
> 1.8/sec     0.733/sec        0.0122/sec   total: 44
> 2017-12-14T23:37:08.708Z|00025|coverage|INFO|pstream_open              
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00026|coverage|INFO|stream_open               
> 0.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00027|coverage|INFO|util_xalloc             
> 182.6/sec   121.383/sec        2.0231/sec   total: 7283
> 2017-12-14T23:37:08.708Z|00028|coverage|INFO|netdev_get_hwaddr         
> 0.4/sec     0.033/sec        0.0006/sec   total: 2
> 2017-12-14T23:37:08.708Z|00029|coverage|INFO|netlink_received          
> 0.6/sec     0.050/sec        0.0008/sec   total: 3
> 2017-12-14T23:37:08.708Z|00030|coverage|INFO|netlink_sent               0
> errs=5703523.2/sec     0.017/sec        0.0003/sec   total: 1
> 2017-12-14T23:37:08.708Z|00031|coverage|INFO|86 events never hit
> 2017-12-14T23:37:08.710Z|00032|poll_loop|INFO|wakeup due to [POLLIN] on fd
> 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98%
> CPU usage)
> 2017-12-14T23:37:08.710Z|00033|memory|INFO|59108 kB peak resident set size
> after 40.7 seconds
> 2017-12-14T23:37:08.710Z|00034|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.7.0
> 2017-12-14T23:37:14.753Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports recirculation
> 2017-12-14T23:37:14.753Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN
> header stack length probed as 1
> 2017-12-14T23:37:14.753Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
> label stack length probed as 3
> 2017-12-14T23:37:14.753Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports truncate action
> 2017-12-14T23:37:14.753Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports unique flow ids
> 2017-12-14T23:37:14.753Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_state
> 2017-12-14T23:37:14.753Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_zone
> 2017-12-14T23:37:14.753Z|00042|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_mark
> 2017-12-14T23:37:14.753Z|00043|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> supports ct_label
> 2017-12-14T23:37:14.753Z|00044|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath
> does not support ct_state_nat
> 2017-12-14T23:37:14.762Z|00045|bridge|INFO|bridge br0: added interface br0
> on port 65534
> 2017-12-14T23:37:14.762Z|00046|bridge|INFO|bridge br0: using datapath ID
> 0000ca98fcc4fa43
> 2017-12-14T23:37:14.762Z|00047|connmgr|INFO|br0: added service controller
> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
> 2017-12-14T23:37:22.782Z|00048|netdev_dpdk|WARN|Rx checksum offload is not
> supported on device 0
> 2017-12-14T23:37:22.788Z|00049|dpif_netdev|INFO|Created 2 pmd threads on
> numa node 0
> 2017-12-14T23:37:23.048Z|00050|netdev_dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
> 2017-12-14T23:37:23.049Z|00051|bridge|INFO|bridge br0: added interface
> dpdkport0 on port 1
> 2017-12-14T23:37:23.049Z|00052|bridge|INFO|bridge br0: using datapath ID
> 00009cdc71b3b8a0
> 2017-12-14T23:37:29.228Z|00053|memory|INFO|peak resident set size grew 89%
> in last 20.5 seconds, from 59108 kB to 111796 kB
> 2017-12-14T23:37:29.229Z|00054|memory|INFO|handlers:1 ports:2 revalidators:1
> rules:5
> 2017-12-14T23:37:31.209Z|00055|netdev_dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
> 2017-12-14T23:37:31.209Z|00056|bridge|INFO|bridge br0: added interface
> dpdkport1 on port 2
> 2017-12-14T23:37:49.367Z|00057|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 deletes)
> 2017-12-14T23:37:54.376Z|00058|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 adds)
> 2017-12-14T23:37:54.378Z|00059|connmgr|INFO|br0<->unix: 1 flow_mods in the
> last 0 s (1 adds)
> 
> 
> [root@en104 openvswitch-2.7.0]# ./utilities/ovs-vsctl show
> 9f085008-4b7d-401d-b7f9-1a4399324c4c
>     Bridge "br0"
>         Port "br0"
>             Interface "br0"
>                 type: internal
>         Port "dpdkport1"
>             Interface "dpdkport1"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:12:00.1", n_rxq="1",
> n_rxq_desc="2048"}
>         Port "dpdkport0"
>             Interface "dpdkport0"
>                 type: dpdk
>                 options: {dpdk-devargs="0000:12:00.0", n_rxq="1",
> n_rxq_desc="2048"}
>     ovs_version: "2.7.0"
> 
> 
> 
> => Generated Spirent traffic:
> 
> # ovs-ofctl dump-flows br0
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=3286.471s, table=0, n_packets=633017838,
> n_bytes=37981070280, idle_age=0, in_port=1 actions=output:2
>  cookie=0x0, duration=3286.470s, table=0, n_packets=694627989,
> n_bytes=41677679340, idle_age=0, in_port=2 actions=output:1

Wow! Seems like you've got it working fine with this version of OVS. 
Is there a problem show-casing this for your POC ? 
 In any case like i said earlier even with OVS 2.6 , you got it pointing to the right version of DPDK , BNXT PMD driver loaded and correctly detected the bnxt ports/devices on your setup. 
Now, whatever problem you are seeing seems like some OVS related problem and not sure how much we can do to help here
I've already requested Ajit to set it up so he can possibly guide you runtime in his timezone

Thanks
Som

Comment 70 Dilip Daya 2017-12-15 14:37:53 UTC
(In reply to Ajit Khaparde from comment #61)
> FYI..
> No need to separately patch the 16.11 code with the PCI IDs anymore.
> The patch with the PCI IDs has been applied to the 16.11 stable tree
> with commit 6c2d431164f3d837.
> 
> ----
> commit 6c2d431164f3d8374fc9ce4396746495b46cab95 (HEAD, origin/16.11)
> Author: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Date:   Tue Dec 12 11:39:00 2017 -0800
> 
>     net/bnxt: support new PCI IDs
>     
>     [ upstream commit 1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 ]
>     
>     Add support for PCI IDs which was removed as a part of
>     commit 0e7bd0b2d6a5 ("net/bnxt: remove support for few devices")
>     
>     Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>     Acked-by: Luca Boccassi <bluca@debian.org>

Hi Ajit,

[root@en104 dpdk-16.11.4-ajit]# make install DESTDIR=dpdk-install T=x86_64-native-linuxapp-gcc
Configuration done
== Build lib
== Build lib/librte_compat
  SYMLINK-FILE include/rte_compat.h
== Build lib/librte_eal
== Build lib/librte_eal/common
  SYMLINK-FILE include/generic/rte_atomic.h
...
...
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2295:2: error: too few arguments to function ‘ndo_dflt_bridge_getlink’
  return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
  ^
In file included from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/dst.h:13:0,
                 from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/net/sock.h:72,
                 from /usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/tcp.h:23,
                 from /home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:34:
/usr/src/kernels/3.10.0-693.11.1.el7.x86_64/include/linux/rtnetlink.h:115:12: note: declared here
 extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
            ^
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: At top level:
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2324:2: error: unknown field ‘ndo_set_vf_vlan’ specified in initializer
  .ndo_set_vf_vlan = igb_ndo_set_vf_vlan,
  ^
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c: In function ‘igb_ndo_bridge_getlink’:
/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2303:1: error: control reaches end of non-void function [-Werror=return-type]
 }
 ^
cc1: all warnings being treated as errors
make[10]: *** [/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o] Error 1
make[9]: *** [_module_/home/dilip/dpdk_upstream/dpdk-16.11.4-ajit/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni] Error 2
make[8]: *** [sub-make] Error 2
make[7]: *** [rte_kni.ko] Error 2
make[6]: *** [kni] Error 2
make[5]: *** [linuxapp] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2


=> Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in 
   config/common_linuxapp and that produced a successful build:

...
Build complete [x86_64-native-linuxapp-gcc]
================== Installing dpdk-install/
Installation in dpdk-install/ complete


=> Would this be fixed by RH ?

Comment 71 Dilip Daya 2017-12-15 14:52:54 UTC
(In reply to Somnath Kotur from comment #69)
> (In reply to Dilip Daya from comment #68)
...
...
> > => Generated Spirent traffic:
> > 
> > # ovs-ofctl dump-flows br0
> > NXST_FLOW reply (xid=0x4):
> >  cookie=0x0, duration=3286.471s, table=0, n_packets=633017838,
> > n_bytes=37981070280, idle_age=0, in_port=1 actions=output:2
> >  cookie=0x0, duration=3286.470s, table=0, n_packets=694627989,
> > n_bytes=41677679340, idle_age=0, in_port=2 actions=output:1
> 
> Wow! Seems like you've got it working fine with this version of OVS. 
> Is there a problem show-casing this for your POC ? 

Problem would be supportability of these stacked components:

* RHOSP 10
* openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm
* dpdk-stable-16.11.x version (I'm assuming dpdk-stable-16.11.5 with fixes by RH for comment #70)
* RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)

=> Question: is the above stack supportable by RH and Broadcom?

Comment 72 Sean Merrow 2017-12-15 15:06:24 UTC
OSP 10 currently does not support OVS 2.7, so engineering's suggestion is as follows:

My suggestion is to do the following for the 2.6.1 for OSP10, DPDK 16.11.

- Get the source RPM, for Red Hat OpenStack 10 this is
openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
- Install the RPM
- Modify the spec file, and replace the DPDK tar.xz
- Build the RPM from here, and use it for the PVP test [1].

[1] https://github.com/ctrautma/RHEL_NIC_QUALIFICATION

Comment 73 Dilip Daya 2017-12-15 15:20:21 UTC
(In reply to Sean Merrow from comment #72)
> OSP 10 currently does not support OVS 2.7, so engineering's suggestion is as
> follows:
> 
> My suggestion is to do the following for the 2.6.1 for OSP10, DPDK 16.11.
> 
> - Get the source RPM, for Red Hat OpenStack 10 this is
> openvswitch-2.6.1-16.git20161206.el7ost.src.rpm

Already tested.  Please review comment #62 for failures from using: openvswitch-2.6.1-16.git20161206.el7ost and needs to be fixed:

2017-12-13T20:54:42.150Z|00047|bridge|WARN|could not open network device dpdkport0 (No such device)


Thanks,
-DilipD.

Comment 74 Sean Merrow 2017-12-15 15:34:08 UTC
Okay, let me make sure I've got this right. Please confirm the following:

WORKS
======
- OSP 10
- OVS 2.7.0-8 from the FDP channel
- DPDK 16.11-x with fixes
- RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)

DOES NOT WORK
=============
- OSP 10
- OVS 2.6.1 shipped with OSP 10
- DPDK 16.11-x with fixes
- RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)

So the only difference between the above is the version of OVS?

Thanks,
Sean

Comment 75 Dilip Daya 2017-12-15 15:52:28 UTC
(In reply to Sean Merrow from comment #74)
> Okay, let me make sure I've got this right. Please confirm the following:
> 
> WORKS
> ======
> - OSP 10
> - OVS 2.7.0-8 from the FDP channel
> - DPDK 16.11-x with fixes
> - RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)

Correct, I tested and dpdk traffic flows (comment #68) -- WORKS
- OVS 2.7.0-8 from the FDP channel
- DPDK 16.11-x with fixes
- RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)


> DOES NOT WORK
> =============
> - OSP 10
> - OVS 2.6.1 shipped with OSP 10
> - DPDK 16.11-x with fixes
> - RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> 
> So the only difference between the above is the version of OVS?

Correct--OVS 2.6.1 shipped with OSP 10 fails (comment #62) -- DOES NOT WORK

Comment 76 Aaron Conole 2017-12-17 17:16:17 UTC
Please reread my comment.  Port naming changed in OvS 2.7.  With Open vSwitch 2.6, you *MUST* use name 'dpdk{0,1,2,...}' for the port, and that port *MUST* correspond to the correct dpdk port number.  The argument 'dpdk-devargs=' is *NOT* intepreted by OvS 2.6.

For more information, see:
https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md

Did you try the above and still see errors?

Comment 77 Dilip Daya 2017-12-18 17:40:43 UTC
(In reply to Aaron Conole from comment #76)
> Please reread my comment.  Port naming changed in OvS 2.7.  With Open
> vSwitch 2.6, you *MUST* use name 'dpdk{0,1,2,...}' for the port, and that
> port *MUST* correspond to the correct dpdk port number.  The argument
> 'dpdk-devargs=' is *NOT* intepreted by OvS 2.6.
> 
> For more information, see:
> https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md
> 
> Did you try the above and still see errors?

Hi Aaron,

I went back to creating a clean slate and recompiled all software to make progress with the following caveat:

Environment (WORKS)
-------------------

* RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
* openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
  - Enabled CONFIG_RTE_LIBRTE_BNXT_PMD in openvswitch.spec
* Modified dpdk tree (dpdk-16.11) within
  openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
  - Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
    config/common_linuxapp
  - Applied patch from:
    - http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415

[1] OVS startup:

BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.

# cat /usr/local/var/log/openvswitch/ovs-vswitchd.log
...
2017-12-17T23:01:20.362Z|00045|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-17T23:01:20.362Z|00046|bridge|INFO|bridge br0: using datapath ID 00002e82272a014c
2017-12-17T23:01:20.362Z|00047|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-17T23:01:28.367Z|00048|netdev_dpdk|WARN|'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported
2017-12-17T23:01:28.367Z|00049|netdev|WARN|dpdk0: could not set configuration (Invalid argument)


=> Modified OVS startup script to have:

$OVS_UTILS/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=$P1
$OVS_UTILS/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk options:dpdk-devargs=$P2


++ /root/rpmbuild_openvswitch-2.6.1-16.git20161206.el7ost_working_dpdk/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:12:00.0
++ sleep 8
++ /root/rpmbuild_openvswitch-2.6.1-16.git20161206.el7ost_working_dpdk/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk options:dpdk-devargs=0000:12:00.1


[root@en104 rpmbuild]# cat /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-18T16:12:41.765Z|00001|vlog|INFO|opened log file /usr/local/var/log/openvswitch/ovs-vswitchd.log
2017-12-18T16:12:41.769Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0
2017-12-18T16:12:41.769Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1
2017-12-18T16:12:41.769Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores
2017-12-18T16:12:41.769Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
2017-12-18T16:12:41.769Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
2017-12-18T16:12:41.770Z|00007|dpdk|INFO|DPDK Enabled - initializing...
2017-12-18T16:12:41.771Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /usr/local/var/run/openvswitch
2017-12-18T16:12:41.771Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 1 --socket-mem 2048,0
2017-12-18T16:13:21.895Z|00010|dpdk|INFO|DPDK Enabled - initialized
2017-12-18T16:13:21.896Z|00011|timeval|WARN|Unreasonably long 40125ms poll interval (49ms user, 39477ms system)
2017-12-18T16:13:21.896Z|00012|timeval|WARN|faults: 82161 minor, 0 major
2017-12-18T16:13:21.896Z|00013|timeval|WARN|disk: 0 reads, 8 writes
2017-12-18T16:13:21.896Z|00014|timeval|WARN|context switches: 5 voluntary, 5 involuntary
2017-12-18T16:13:21.896Z|00015|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour,  hash=fde89638:
2017-12-18T16:13:21.896Z|00016|coverage|INFO|bridge_reconfigure         0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00017|coverage|INFO|cmap_expand                0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-18T16:13:21.896Z|00018|coverage|INFO|miniflow_malloc            0.4/sec     0.150/sec        0.0025/sec   total: 9
2017-12-18T16:13:21.896Z|00019|coverage|INFO|hmap_pathological          0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00020|coverage|INFO|hmap_expand               10.2/sec     6.333/sec        0.1056/sec   total: 380
2017-12-18T16:13:21.896Z|00021|coverage|INFO|txn_unchanged              0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-18T16:13:21.896Z|00022|coverage|INFO|txn_incomplete             0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00023|coverage|INFO|poll_create_node           1.0/sec     0.667/sec        0.0111/sec   total: 40
2017-12-18T16:13:21.896Z|00024|coverage|INFO|seq_change                 1.8/sec     0.733/sec        0.0122/sec   total: 44
2017-12-18T16:13:21.896Z|00025|coverage|INFO|pstream_open               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00026|coverage|INFO|stream_open                0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00027|coverage|INFO|util_xalloc              182.6/sec   121.383/sec        2.0231/sec   total: 7283
2017-12-18T16:13:21.896Z|00028|coverage|INFO|netdev_get_hwaddr          0.4/sec     0.033/sec        0.0006/sec   total: 2
2017-12-18T16:13:21.896Z|00029|coverage|INFO|netlink_received           0.6/sec     0.050/sec        0.0008/sec   total: 3
2017-12-18T16:13:21.896Z|00030|coverage|INFO|netlink_sent               0.2/sec     0.017/sec        0.0003/sec   total: 1
2017-12-18T16:13:21.896Z|00031|coverage|INFO|86 events never hit
2017-12-18T16:13:21.898Z|00032|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (<->/usr/local/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 (98% CPU usage)
2017-12-18T16:13:21.898Z|00033|memory|INFO|59108 kB peak resident set size after 40.1 seconds
2017-12-18T16:13:21.898Z|00034|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.7.0
2017-12-18T16:13:28.032Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-12-18T16:13:28.032Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1
2017-12-18T16:13:28.032Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-12-18T16:13:28.032Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action
2017-12-18T16:13:28.032Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-12-18T16:13:28.032Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state
2017-12-18T16:13:28.032Z|00041|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone
2017-12-18T16:13:28.032Z|00042|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark
2017-12-18T16:13:28.032Z|00043|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label
2017-12-18T16:13:28.032Z|00044|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state_nat
2017-12-18T16:13:28.041Z|00045|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-18T16:13:28.041Z|00046|bridge|INFO|bridge br0: using datapath ID 0000a6f515f72b4b
2017-12-18T16:13:28.041Z|00047|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-18T16:13:36.053Z|00048|netdev_dpdk|WARN|Rx checksum offload is not supported on device 0
2017-12-18T16:13:36.059Z|00049|dpif_netdev|INFO|Created 2 pmd threads on numa node 0
2017-12-18T16:13:36.319Z|00050|netdev_dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-18T16:13:36.319Z|00051|bridge|INFO|bridge br0: added interface dpdk0 on port 1
2017-12-18T16:13:36.320Z|00052|bridge|INFO|bridge br0: using datapath ID 00009cdc71b3b8a0
2017-12-18T16:13:42.433Z|00053|memory|INFO|peak resident set size grew 89% in last 20.5 seconds, from 59108 kB to 111800 kB
2017-12-18T16:13:42.433Z|00054|memory|INFO|handlers:1 ports:2 revalidators:1 rules:5
2017-12-18T16:13:44.474Z|00055|netdev_dpdk|INFO|Port 1: 9c:dc:71:b3:b8:a8
2017-12-18T16:13:44.475Z|00056|bridge|INFO|bridge br0: added interface dpdk1 on port 2
2017-12-18T16:14:02.735Z|00057|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 deletes)
2017-12-18T16:14:07.737Z|00058|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-12-18T16:14:07.739Z|00059|connmgr|INFO|br0<->unix: 1 flow_mods in the last 0 s (1 adds)

[root@en104 rpmbuild]# export OVS_UTILS=/root/rpmbuild/BUILD/openvswitch-2.6.1/utilities

[root@en104 rpmbuild]# $OVS_UTILS/ovs-ctl -V
/root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-ctl (Open vSwitch) 2.6.2

[root@en104 rpmbuild]# $OVS_UTILS/ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.6.2
DB Schema 7.14.0

[root@en104 rpmbuild]# $OVS_UTILS/ovs-ofctl -V
ovs-ofctl (Open vSwitch) 2.6.2
OpenFlow versions 0x1:0x4

[root@en104 rpmbuild]# $OVS_UTILS/ovs-appctl -V
ovs-appctl (Open vSwitch) 2.6.2



=> Start Spirent traffic flows:

[root@en104 rpmbuild]# BUILD/openvswitch-2.6.1/utilities/ovs-ofctl dump-flows br0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=114.780s, table=0, n_packets=38233146, n_bytes=2293988760, idle_age=0, in_port=1 actions=output:2
 cookie=0x0, duration=114.779s, table=0, n_packets=38233157, n_bytes=2293989420, idle_age=0, in_port=2 actions=output:1

Comment 78 Dilip Daya 2017-12-19 15:26:30 UTC

(In reply to Dilip Daya from comment #77)
> (In reply to Aaron Conole from comment #76)
> > Please reread my comment.  Port naming changed in OvS 2.7.  With Open
> > vSwitch 2.6, you *MUST* use name 'dpdk{0,1,2,...}' for the port, and that
> > port *MUST* correspond to the correct dpdk port number.  The argument
> > 'dpdk-devargs=' is *NOT* intepreted by OvS 2.6.
> > 
> > For more information, see:
> > https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK.md
> > 
> > Did you try the above and still see errors?
> 
> Hi Aaron,
> 
> I went back to creating a clean slate and recompiled all software to make
> progress with the following caveat:
> 
> Environment (WORKS)
> -------------------
> 
> * RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
> * openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
>   - Enabled CONFIG_RTE_LIBRTE_BNXT_PMD in openvswitch.spec
> * Modified dpdk tree (dpdk-16.11) within
>   openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
>   - Disabled CONFIG_RTE_LIBRTE_KNI and CONFIG_RTE_KNI_KMOD in
>     config/common_linuxapp
>   - Applied patch from:
>     -
> http://www.dpdk.org/browse/dpdk/commit/
> ?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415


Just an FYI regarding the DPDK patch for BRCM NIC:

- Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
  Subsystem: Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28 Adapter

patch from Broadcom:
- http://www.dpdk.org/browse/dpdk/commit/?id=1f4a84672ecbd28bfaaa77f99b6e5b9b77896415 was submitted upstream to
dpdk-stable tree:
http://dpdk.org/browse/dpdk-stable/log/?h=16.11 and should expect a
dpdk-stable-16.11.5 release sometime March-2018.

Comment 79 Aaron Conole 2017-12-19 15:39:08 UTC
I'm confused - the logs there show ovs 2.7.0 - are you sure you started the correct version?  dpdk-devargs wasn't added until 2.7.0 (as part of commit 55e075e65ef9 "netdev-dpdk: Arbitrary 'dpdk' port naming"), so getting an error message as that would indicate a problem.

Comment 80 Dilip Daya 2017-12-19 16:34:10 UTC
(In reply to Aaron Conole from comment #79)
> I'm confused - the logs there show ovs 2.7.0 - are you sure you started the
> correct version?  dpdk-devargs wasn't added until 2.7.0 (as part of commit
> 55e075e65ef9 "netdev-dpdk: Arbitrary 'dpdk' port naming"), so getting an
> error message as that would indicate a problem.

You are correct---I did not do "make install" for OVS to overwrite older remnants of ovs-2.7, once that was done it fails when using:

+ /root/rpmbuild/BUILD/openvswitch-2.6.1/dpdk-16.11/tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:12:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:12:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:12:00.0
ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.

=> /usr/local/var/log/openvswitch/ovs-vswitchd.log:
...
2017-12-19T16:22:54.885Z|00042|bridge|INFO|bridge br0: added interface br0 on port 65534
2017-12-19T16:22:54.885Z|00043|bridge|INFO|bridge br0: using datapath ID 00009626b482af40
2017-12-19T16:22:54.885Z|00044|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-19T16:22:58.894Z|00045|memory|INFO|peak resident set size grew 53% in last 10.1 seconds, from 59688 kB to 91416 kB
2017-12-19T16:22:58.894Z|00046|memory|INFO|handlers:1 ports:1 revalidators:1 rules:5
2017-12-19T16:23:03.150Z|00047|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-19T16:23:03.533Z|00048|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-19T16:23:03.533Z|00049|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
...



=> then modified as follows:

++ /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.


=> /usr/local/var/log/openvswitch/ovs-vswitchd.log:
...
2017-12-19T16:30:02.160Z|00043|bridge|INFO|bridge br0: using datapath ID 00004e4a7e3dc54d
2017-12-19T16:30:02.161Z|00044|connmgr|INFO|br0: added service controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
2017-12-19T16:30:06.171Z|00045|memory|INFO|peak resident set size grew 52% in last 10.1 seconds, from 60600 kB to 92252 kB
2017-12-19T16:30:06.171Z|00046|memory|INFO|handlers:1 ports:1 revalidators:1 rules:5
2017-12-19T16:30:10.430Z|00047|dpdk|INFO|Port 0: 9c:dc:71:b3:b8:a0
2017-12-19T16:30:10.778Z|00048|dpdk|ERR|Interface dpdk0 start error: Unknown error -4
2017-12-19T16:30:10.779Z|00049|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Unknown error -4
...


[root@en104 rpmbuild]# /root/rpmbuild/BUILD/openvswitch-2.6.1/utilities/ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.6.2
DB Schema 7.14.0

Comment 81 Andy Gospodarek 2018-01-09 21:49:31 UTC
If I'm reading things correctly it appears that things may not be working as smoothly as HPE hoped.  Was there ever a resolution to this?

I've been taking a look at the DPDK and OVS+DPDK bugzillas Broadcom had opened and just wanted to provide some clarification going forward.

The latest DPDK Stable Tree for 16.11[1] contains support for bnxt PCI IDs[2] that should satisfy most mutual customer requests.  It also has supported a VFIO backend since it was included upstream as a supported PMD.  We are requesting that this is pulled into the Open vSwitch package for RHEL7 Fast Datapath channel.

We are currently working to gather full PVP test results with the expectation that having those results will result in support for bnxt PMD in OVS+DPDK included in the Fast Datapath Channel.  The team had some issues getting TRex running since it also did not include support for the bnxt PMD, but the team was able to use it with another vendor's adapter as the transmitter.  Tests show an increase in throughput between the L3 OVS+DPDK datapath and L3 OVS+kernel datapath, so this is a positive sign.

Thanks!

1. http://dpdk.org/browse/dpdk-stable/log/?h=16.11
2. http://dpdk.org/browse/dpdk-stable/commit/?h=16.11&id=6c2d431164f3d8374fc9ce4396746495b46cab95

Comment 82 Dilip Daya 2018-01-10 17:58:45 UTC
(In reply to Andy Gospodarek from comment #81)
> If I'm reading things correctly it appears that things may not be working as
> smoothly as HPE hoped.  Was there ever a resolution to this?
> 
> I've been taking a look at the DPDK and OVS+DPDK bugzillas Broadcom had
> opened and just wanted to provide some clarification going forward.
> 
> The latest DPDK Stable Tree for 16.11[1] contains support for bnxt PCI
> IDs[2] that should satisfy most mutual customer requests.  It also has
> supported a VFIO backend since it was included upstream as a supported PMD. 
> We are requesting that this is pulled into the Open vSwitch package for
> RHEL7 Fast Datapath channel.

16.11 [1] will _not_ compile with RHEL7.4 kernel 3.10.0-693.11.1.el7.x86_64 as it's not compatible with igb driver especially the ".ndo_bridge_* " routines.  Modifying x86_64-native-linuxapp-gcc/.config and setting these 2 config variables to 'n') provides a successful compilation:

    - CONFIG_RTE_LIBRTE_KNI=n
    - CONFIG_RTE_KNI_KMOD=n

1. http://dpdk.org/browse/dpdk-stable/log/?h=16.11

 
> We are currently working to gather full PVP test results with the
> expectation that having those results will result in support for bnxt PMD in
> OVS+DPDK included in the Fast Datapath Channel.


    WORKS
    =====

    - RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)
    - DPDK-16.11 [1] http://dpdk.org/browse/dpdk-stable/log/?h=16.11
      with compilation fixes noted above.
    - openvswitch-2.7.0-8.git20170530.el7fdb.src.rpm from the FDP channel


    DOES NOT WORK
    =============

    - openvswitch-2.6.1-16.git20161206.el7ost.src.rpm
    - DPDK-16.11 [1] http://dpdk.org/browse/dpdk-stable/log/?h=16.11
      with compilation fixes noted above.
    - RHEL 7.4 (3.10.0-693.11.1.el7.x86_64)


Thanks,
-DilipD.

Comment 83 Dilip Daya 2018-01-19 17:37:21 UTC
Created attachment 1383532 [details]
rhel-7-fast-datapath-htb-rpms_failure_report

Testing packages from repo id: rhel-7-fast-datapath-htb-rpms/x86_64

[root@en104 ~]# rpm -qa | grep dpdk
dpdk-doc-17.05-2.el7fdb.noarch
dpdk-17.05-2.el7fdb.x86_64
dpdk-tools-17.05-2.el7fdb.x86_64
dpdk-devel-17.05-2.el7fdb.x86_64

[root@en104 ~]# rpm -qa | grep openvswitch
openvswitch-2.8.0-4.el7fdb.x86_64
python-openvswitch-2.8.0-4.el7fdb.noarch
openvswitch-devel-2.8.0-4.el7fdb.x86_64
openvswitch-test-2.8.0-4.el7fdb.noarch

=> [root@en104 ~]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:12:00.0

=> dmesg:
[ 2481.008509] pmd13[6209]: segfault at 50 ip 000055bb7f6c2ebb sp 00007fd4ea7fb690 error 4
[ 2481.008551] ovs-vswitchd[5398]: segfault at 50 ip 000055bb7f6c3851 sp 00007ffc348d0628 error 4
[ 2481.008553]  in ovs-vswitchd[55bb7f520000+4d8000]
[ 2481.028259]  in ovs-vswitchd[55bb7f520000+4d8000]
[ 2521.379095] vfio-pci 0000:12:00.0: enabling device (0400 -> 0402)
[ 2521.485707] vfio_ecap_init: 0000:12:00.0 hiding ecap 0x19@0x300
[ 2521.491653] vfio_ecap_init: 0000:12:00.0 hiding ecap 0x1f@0x200
[ 2521.601477] vfio-pci 0000:12:00.1: enabling device (0400 -> 0402)
[ 2522.224788] pmd6[6879]: segfault at 50 ip 000055bb7f6c2ebb sp 00007fd4f1ffa690 error 4 in ovs-vswitchd[55bb7f520000+4d8000]
[ 2562.414726] vfio-pci 0000:12:00.0: enabling device (0400 -> 0402)
[ 2562.521688] vfio_ecap_init: 0000:12:00.0 hiding ecap 0x19@0x300
[ 2562.527632] vfio_ecap_init: 0000:12:00.0 hiding ecap 0x1f@0x200
[ 2562.637459] vfio-pci 0000:12:00.1: enabling device (0400 -> 0402)
[ 2563.049361] pmd11[7561]: segfault at 50 ip 000055bb7f6c2ebb sp 00007fd4f2ffc690 error 4
[ 2563.049411] [6904]: segfault at 50 ip 000055bb7f6c3851 sp 00007ffc348d0628 error 4
[ 2563.049413]  in ovs-vswitchd[55bb7f520000+4d8000]
[ 2563.068108]  in ovs-vswitchd[55bb7f520000+4d8000]
...

See attached: rhel-7-fast-datapath-htb-rpms_failure_report.txt

Comment 84 Sean Merrow 2018-02-06 16:32:24 UTC
Hi Andy,

Do you have an update on the PVP testing?

Thanks,
Sean

Comment 85 Andy Gospodarek 2018-02-20 15:17:17 UTC
Sean, I think the PVP test results are being shared by Lee Reed and Vinay Gupta and others and results have been uploaded to bug 1537302.

Comment 86 Trinh Dao 2018-02-28 18:45:10 UTC
Sean, is this bug going to be fix in any of the RHEL7.5 build?

Comment 87 Trinh Dao 2018-03-09 16:51:04 UTC
Monte, 
I do not have access to bug 1537302.  Please grant HPE access to bug 1537302.

Comment 88 Trinh Dao 2018-03-14 12:29:38 UTC
Hi Andy,
What is the status on the PVP test? HPE Mushtaq needs update.

Thank you,
trinh

Comment 89 Andy Gospodarek 2018-03-26 12:57:47 UTC
Trinh, overall I think the results were positive.  We had a few issues with some tests and brought those up along with the results in bug 1537302 (which I see you should be able to access).

Comment 90 Rashid Khan 2018-04-13 18:47:24 UTC
Andy, 
Do we need anything more than what is FDP 1804 release?
Can you please ask Brcm on-site to please check. Sorry I forgot the gentleman's name.

Comment 93 Davide Caratti 2018-04-20 12:43:14 UTC
hello,

(In reply to Andy Gospodarek from comment #89)
> Trinh, overall I think the results were positive.  We had a few issues with
> some tests and brought those up along with the results in bug 1537302 (which
> I see you should be able to access).

(In reply to Rashid Khan from comment #90)
> Andy, 
> Do we need anything more than what is FDP 1804 release?

AFAIU this bz tracks inclusion in fast-datapath-beta-rhel7.

FDB rpms are usable on NetXtreme NICS since openvswitch-2.9.0-2.el7fdb, so I changed state as CLOSED CURRENTRELEASE.

regards,
-- 
davide

Comment 94 Dilip Daya 2018-04-20 15:02:27 UTC
Hi Davide:

I do not see "openvswitch-2.9.0-2.el7fdb" listed in repo: "rhel-7-fast-datapath-htb-rpms":

# yum repo-pkgs rhel-7-fast-datapath-htb-rpms list

Loaded plugins: auto-update-debuginfo, langpacks, search-disabled-repos
Installed Packages
driverctl.noarch                 1:0.95-1.el7fdb   @rhel-7-fast-datapath-htb-rpms
Available Packages
dpdk.x86_64                      1:17.05-2.el7fdb  rhel-7-fast-datapath-htb-rpms 
dpdk-devel.x86_64                1:17.05-2.el7fdb  rhel-7-fast-datapath-htb-rpms 
dpdk-doc.noarch                  1:17.05-2.el7fdb  rhel-7-fast-datapath-htb-rpms 
dpdk-tools.x86_64                1:17.05-2.el7fdb  rhel-7-fast-datapath-htb-rpms 
kmod-vfio.x86_64                 0.3-3.el7fdb      rhel-7-fast-datapath-htb-rpms 
openvswitch.x86_64               1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-devel.x86_64         1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-ovn-central.x86_64   1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-ovn-common.x86_64    1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-ovn-docker.x86_64    1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-ovn-host.x86_64      1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-ovn-vtep.x86_64      1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
openvswitch-test.noarch          1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms 
python-openvswitch.noarch        1:2.8.0-4.el7fdb  rhel-7-fast-datapath-htb-rpms


Thanks,
-DilipD.

Comment 95 Davide Caratti 2018-04-20 16:14:11 UTC
(In reply to Dilip Daya from comment #94)
> Hi Davide:
> 
> I do not see "openvswitch-2.9.0-2.el7fdb" listed in repo:
> "rhel-7-fast-datapath-htb-rpms":
> 
> # yum repo-pkgs rhel-7-fast-datapath-htb-rpms list
> 

Ops! you are right, 2.9.0-2 is part of a RHBA which hasn't been published yet.
Setting the state back to MODIFIED, so that this bz becomes eligible.

> Loaded plugins: auto-update-debuginfo, langpacks, search-disabled-repos
> Installed Packages
> driverctl.noarch                 1:0.95-1.el7fdb  
> @rhel-7-fast-datapath-htb-rpms
> Available Packages
> dpdk.x86_64                      1:17.05-2.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> dpdk-devel.x86_64                1:17.05-2.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> dpdk-doc.noarch                  1:17.05-2.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> dpdk-tools.x86_64                1:17.05-2.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> kmod-vfio.x86_64                 0.3-3.el7fdb     
> rhel-7-fast-datapath-htb-rpms 
> openvswitch.x86_64               1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-devel.x86_64         1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-ovn-central.x86_64   1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-ovn-common.x86_64    1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-ovn-docker.x86_64    1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-ovn-host.x86_64      1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-ovn-vtep.x86_64      1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> openvswitch-test.noarch          1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms 
> python-openvswitch.noarch        1:2.8.0-4.el7fdb 
> rhel-7-fast-datapath-htb-rpms
> 
> 
> Thanks,
> -DilipD.

Comment 96 Joseph Kachuck 2018-08-07 15:12:33 UTC
Hello,
Please confirm I see the current released version of this package is:
 openvswitch-2.9.0-47.el7fdp

Please confirm if this BZ maybe able to be closed.

Thank You
Joe Kachuck

Comment 97 Dilip Daya 2018-08-20 13:10:15 UTC
Hi Joe:

Yes, this BZ can now be closed as 2.9.0-47.el7fdp.2 is now available/installed.

# yum list --showduplicates openvswitch
Loaded plugins: auto-update-debuginfo, langpacks, product-id, search-disabled-repos, subscription-manager
Installed Packages
openvswitch.x86_64                                                    2.9.0-47.el7fdp.2                                                    installed                      
Available Packages
openvswitch.x86_64                                                    2.0.0-7.el7                                                          rhel-7-server-optional-rpms    
openvswitch.x86_64                                                    2.9.0-19.el7fdp.1                                                    rhel-7-server-openstack-13-rpms


Thanks,
-DilipD.

Comment 98 Davide Caratti 2018-08-20 13:21:20 UTC
(In reply to Joseph Kachuck from comment #96)
> Hello,
> Please confirm I see the current released version of this package is:
>  openvswitch-2.9.0-47.el7fdp
> 
> Please confirm if this BZ maybe able to be closed.
> 
> Thank You
> Joe Kachuck

(In reply to Dilip Daya from comment #97)
> Hi Joe:
> 
> Yes, this BZ can now be closed as 2.9.0-47.el7fdp.2 is now
> available/installed.

thanks for checking.
Closing this BZ as CURRENTRELEASE
-- 
davide

Comment 99 Trinh Dao 2018-08-22 13:52:24 UTC
mark HPE verified per comment 97


Note You need to log in before you can comment on or make changes to this bug.