Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1516115

Summary: Test OSP13 with OVS-DPDK + BCM57414 NetXtreme-E NIC
Product: Red Hat Enterprise Linux 7 Reporter: Robin Cernin <rcernin>
Component: openvswitchAssignee: Christian Trautman <ctrautma>
Status: MODIFIED --- QA Contact: ovs-qe <ovs-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.4CC: apevec, atragler, chrisw, fbaudin, jlibosva, mleitner, pvauter, rhos-maint, rkhan, srevivo, sukulkar, tredaelli
Target Milestone: pre-dev-freezeKeywords: TestOnly
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Robin Cernin 2017-11-22 05:25:31 UTC
As per the https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/network_functions_virtualization_planning_guide/#tested_nics the Broadcom NICs are note tested.

And as per http://dpdk.org/doc/nics it is supported.

Environment:
* ProLiant DL380 Gen10
* RHEL 7.4 (3.10.0-693.1.1.el7.x86_64)
* NIC: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
       - Subsystem: Hewlett Packard Enterprise Ethernet 25Gb 2-port 631SFP28 Adapter


# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-693.1.1.el7.x86_64 root=/dev/mapper/rhel_example-root ro crashkernel=auto rd.lvm.lv=rhel_example/root rd.lvm.lv=rhel_example/swap rhgb default_hugepagesz=1GB hugepagesz=1GB hugepages=128 hugepagesz=2M
hugepages=4096 intel_iommu=on iommu=pt nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce idle=poll isolcpus=16-23 nohz_full=16-23 rcu_nocbs=16-23
console=ttyS0,115200n8 selinux=0 enforcing=0


=> Red Hat provided DPDK RPM packages:
# rpm -qa | grep dpdk
dpdk-16.11-3.el7.x86_64
dpdk-tools-16.11-3.el7.x86_64


# lsmod | grep vfio
vfio_pci               41267  0
vfio_iommu_type1       22300  0
vfio                   32367  2 vfio_iommu_type1,vfio_pci irqbypass              13503  2 kvm,vfio_pci


# dpdk-devbind --status

Network devices using DPDK-compatible driver ============================================
0000:86:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en
0000:86:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller' drv=vfio-pci unused=bnxt_en


# lspci -vvs 0000:86:00.0 | grep -i numa
	NUMA node: 1

# taskset 0x10000 /usr/bin/testpmd -c 0xe0000  -n 4 -w 0000:86:00.0 -w 0000:86:00.1 -- -i --rss-ip --nb-cores=2
 --nb-ports=2 --forward-mode=mac --eth-peer=0,00:10:94:10:00:01
 --eth-peer=1,00:10:94:10:00:02 --rxd=4096 --txd=4096 --rxq=1 --txq=1
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
Interactive-mode selected
EAL: Error - exiting with code: 1
  Cause: Invalid port 2



Using upstream dpdk-17.08
=========================

# usertools/dpdk-devbind.py --bind vfio-pci 0000:86:00.0 0000:86:00.1
# usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver ============================================
0000:86:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=
0000:86:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' drv=vfio-pci unused=


# taskset 0x10000 ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe0000  -n 4 -w 0000:86:00.0 -w 0000:86:00.1 -- -i --rss-ip --nb-cores=2
 --nb-ports=2 --forward-mode=mac --eth-peer=0,00:10:94:10:00:01
 --eth-peer=1,00:10:94:10:00:02 --rxd=4096 --txd=4096 --rxq=1 --txq=1
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:86:00.0 on NUMA socket 1
EAL:   probe driver: 14e4:16d7 net_bnxt
EAL:   using IOMMU type 1 (Type 1)
PMD: Broadcom Cumulus driver bnxt

PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.7.7
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem ef610000, node addr 0x7f65b3000000M
EAL: PCI device 0000:86:00.1 on NUMA socket 1
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.8.0:20.6.135
PMD: Driver HWRM version: 1.7.7
PMD: BNXT Driver/HWRM API mismatch.
PMD: Firmware API version is newer than driver.
PMD: The driver may be missing features.
PMD: bnxt found at mem ef600000, node addr 0x7f65b3112000M Interactive-mode selected Set mac packet forwarding mode
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1 Configuring Port 0 (socket 1)
PMD: Port 0 Link Up - speed 25000 Mbps - full-duplex
PMD: VLAN Filtering: 1
PMD: VLAN Strip Offload: 1
Port 0: 9C:DC:71:B3:B8:A0
Configuring Port 1 (socket 1)
PMD: Port 1 Link Up - speed 25000 Mbps - full-duplex
PMD: VLAN Filtering: 1
PMD: VLAN Strip Offload: 1
Port 1: 9C:DC:71:B3:B8:A8
Checking link statuses...
Done
testpmd>

Comment 2 Timothy Redaelli 2017-11-27 15:54:57 UTC
The proper component for this bug should be "dpdk", but unlucky the project "OpenStack" doesn't have it.

Comment 3 Franck Baudin 2017-11-30 13:25:10 UTC
re-routing the BZ to the proper component, NIC certification is a RHEL followed/driven activity, and certification has to be done by the NIC vendor.