Request to add Marvell FastLinQ QL45000 Series QLogic Corp. Vendor-id: 1077 as supported device for SR-IOV with K8s

[hpeaj]

Well known K8s platform is using this operator to allow supported network interfaces to provide SR-IOV capability to K8s Pods
When try this with Marvell Interface (Marvell FastLinQ QL45000 Series QLogic Corp. Vendor-id: 1077 )
Error from server (vendor 1077 is not supported): error when creating "SRIOVoperator/SriovNetworkNodePolicy.yaml": admission webhook "sriov-operator-webhook.sriovnetwork.k8s.cni.cncf.io" denied the request: vendor 1077 is not supported

We have Isolated the challenge to this being the configmap used for the setup of the operator leverage the following input yamls that do not have reference for vendor id 1077 we think following are the yaml inputs that need to modified to support the Marvell interface
• sriov-network-operator/supported-nic-ids_v1_configmap.yaml
• sriov-network-operator/sriov-network-operator_configmap.yaml
• sriov-network-operator/configmap.yaml
• sriov-network-operator/pr-ci-triggers.yml

We have successfully able to provide SR-IOV to POD within Well known K8s platform using Marvell Interface (Marvell FastLinQ QL45000 Series QLogic Corp. Vendor-id: 1077 ) using the https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin approach
Here is some of the changes we made to get this working

1. Created the SR-IOV Network CRD
   $ kubectl create -f deployments/sriov-crd.yaml

Note: - Get the resource name (k8s.v1.cni.cncf.io/resourceName) by describe node as mentioned below

kubectl get node node1 -o json | jq '.status.allocatable'

$ kubectl get node node1 -o json | jq '.status.allocatable'

{
"allocable": {
"cpu": "39430m",
"ephemeral-storage": "86930620273",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"marvell.com/marvell_sriov_dpdk": "3",
"marvell.com/marvell_sriov_netdevice": "3",
"memory": "121944076Ki",
"pods": "250"
}

shown below:
{
"resourceList": [{
"resourceName": "marvell_sriov_netdevice",
"resourcePrefix": "marvell.com",
"selectors": {
"vendors": ["1077"],
"devices": ["1664"],
"drivers": ["qede","qed"]
}
},
{
"resourceName": "marvell_sriov_dpdk",
"resourcePrefix": "marvell.com",
"selectors": {
"vendors": ["1077"],
"devices": ["1664"],
"drivers": ["qede","qed"],
"pfNames": ["ens3f0"],
"needVhostNet": true
}
}

]

}
$ kubectl get node node1 -o json | jq '.status.allocatable'

{
"allocable": {
"cpu": "39430m",
"ephemeral-storage": "86930620273",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"marvell.com/marvell_sriov_dpdk": "3",
"marvell.com/marvell_sriov_netdevice": "3",
"memory": "121944076Ki",
"pods": "250"
}
Deploy test Pod connecting to pre-created SR-IOV network
$ kubectl create -f pod-tc1.yaml
pod "testpod1" created

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
testpod1 1/1 Running 0 3s
Verify Pod network interfaces
$ kubectl exec -it testpod1 -- ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 9a:c0:0e:90:11:04 brd ff:ff:ff:ff:ff:ff
inet 10.56.217.5/8 brd 10.255.255.255 scope global net1
valid_lft forever preferred_lft forever
1246: eth0@if1247: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 26:f2:d1:a6:35:34 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.174/32 scope global eth0
valid_lft forever preferred_lft forever
Verify Pod routing table
$ kubectl exec -it testpod1 -- route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.3.23 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 net1
192.168.3.23 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

kubectl get po -A --kubeconfig $u1 |grep -i multus
kube-system kube-multus-ds-64d9q 1/1 Running 0 2d19h
kube-system kube-multus-ds-c4cnj 1/1 Running 0 2d19h

kubectl get po -A --kubeconfig $u1 |grep -i sriov
default sriovpod1 1/1 Running 0 45h
kube-system kube-sriov-device-plugin-amd64-kxfrg 1/1 Running 0 2d19h
kube-system kube-sriov-device-plugin-amd64-mwqwg 1/1 Running 0 2d19h

sriov-crd-network-attach.yaml.txt
describepod.yaml.txt

We have validated Marvell Interface (Marvell FastLinQ QL45000 Series QLogic Corp. Vendor-id: 1077 ) working with Well known K8s platform able to provide SR-IOV to pods using https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin however is not the official way of doing this on Well known K8s platform

Hence we request your support and guidance on how have could have Marvell Interface (Marvell FastLinQ QL45000 Series QLogic Corp. Vendor-id: 1077 ) as supported network Interface for sriov-network-operator approach

[adrianchiris]

Greetings !

If there are no special configurations required for the NIC to enable and configure SR-IOV then it should be a matter of adding QLogic NICs to the supported-nic-ids config map.

see : https://github.com/k8snetworkplumbingwg/sriov-network-operator/blob/master/doc/quickstart.md#prerequisites

To add it by default, i think it would be preferable that some CI testing is set up to run against QLogic hardware
see:
https://github.com/k8snetworkplumbingwg/sriov-network-operator/tree/master/ci

by editing supported-nic-ids configmap where you able to get sriov-network-operator to work with QLogic NICs ?

You are welcome join Our Working Group meetings (K8s Network & Resource Management).
https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/master/CONTRIBUTING.md#meetings

[hpeaj]

We have successfully tested the Marvell interface using vanilla K8s by editing supported-nic-ids configmap we were able to get sriov-network-operator to work with QLogic NICs
Currently perform the CI testing
Please help us what with list of data that must be collected to the proceed?

[krjagan]

we are trying to validate vendor card using hack/run-e2e-test-kind.sh script with system-service but geeting error.

docker build -t sriov-network-operator:latest -f /root/sriov-network-operator/Dockerfile /root/sriov-network-operator
Sending build context to Docker daemon 191.5 MB
Step 1/9 : FROM golang:1.17 AS builder
Error parsing reference: "golang:1.17 AS builder" is not a valid repository/tag: invalid reference format
Exit code: '1'. Sleeping '10' seconds before retrying
docker build -t sriov-network-operator:latest -f /root/sriov-network-operator/Dockerfile /root/sriov-network-operator
Sending build context to Docker daemon 191.5 MB
Step 1/9 : FROM golang:1.17 AS builder
Error parsing reference: "golang:1.17 AS builder" is not a valid repository/tag: invalid reference format

[root@k8sworker bin]# docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:22 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

[root@k8sworker bin]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

Please provide the process of validating cards.

[zshi-redhat]

we are trying to validate vendor card using hack/run-e2e-test-kind.sh script with system-service but geeting error.

docker build -t sriov-network-operator:latest -f /root/sriov-network-operator/Dockerfile /root/sriov-network-operator Sending build context to Docker daemon 191.5 MB Step 1/9 : FROM golang:1.17 AS builder Error parsing reference: "golang:1.17 AS builder" is not a valid repository/tag: invalid reference format Exit code: '1'. Sleeping '10' seconds before retrying docker build -t sriov-network-operator:latest -f /root/sriov-network-operator/Dockerfile /root/sriov-network-operator Sending build context to Docker daemon 191.5 MB Step 1/9 : FROM golang:1.17 AS builder Error parsing reference: "golang:1.17 AS builder" is not a valid repository/tag: invalid reference format

It looks like an issue with old docker version that doesn't support multi-stage builds?
Possibly a similar issue: kubernetes-sigs/kubebuilder#268

[hpeaj]

[root@k8sworker bin]# docker version
> Client: Docker Engine - Community
Version: 20.10.12

[hpeaj]

As discussed with community attaching the configMap
Will work on the possibility of setting up a CI environment and get contact information for HPE Synergy 6820C 25/50Gb Converged Network Adapter

• For sriov -network-deviceplugin : configMap.yaml.txt
• For sriov-netowrk-operator : supported-nic-ids.yaml.txt

supported-nic-ids.yaml.txt
configMap.yaml.txt

[hpeaj]

The contact information for this card HPE Synergy 6820C 25/50Gb Converged Network Adapter - [email protected]