Can't bind VM interface as DPDK interface inside a k8s POD

[pperiyasamy]

Binding VM's virtio interface (attached as vHost port to OVS running on the Host OS) as DPDK interface into K8S Pod is failing with following error.

"Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "017d716ce70a585a7f23e38aaf463f058d3b396929d791204b9253ed58a27b29" network for pod "samplepod": NetworkPlugin cni failed to set up pod "samplepod_default" network: Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "sriov": failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory"

Here are configuration used for creating the POD:

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: dpdk-conf
spec:
config: '{
"name": "mynet",
"type": "sriov",
"if0": "ens18",
"if0name": "net0",
"dpdk": {
"kernel_driver":"uio",
"dpdk_driver":"igb_uio",
"dpdk_tool":"/usr/src/dpdk-stable-17.11.4/install/share/dpdk/usertools/dpdk-devbind.py"
}
}'
EOF

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: dpdk-conf
spec:
containers:

• name: samplepod
  command: ["/bin/bash", "-c", "sleep 2000000000000"]
  image: dougbtv/centos-network
  EOF

Excerpts from kubectl describe command on the samplepod:

Events:
Type Reason Age From Message

---

Normal Scheduled 8s default-scheduler Successfully assigned default/nginx to ubuntu
Warning FailedCreatePodSandBox 7s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dc4ed48ee816067a2bcb1d9d577cd8304b8183545aee103e19b52175a5093623" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 6s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "9f47783f0743ea5a5f32d4dd75c64506ec7988f6ed2fcc32aa2a300a1737ae4d" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 5s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d3d7c9869e26bb1343dc5f2ab43b786ea002ff4e979f1810f393caef299ff273" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 4s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "49c0f432a088087ff22aeee1a9f4b957d15df88064204d4c90d7839b3f9dea2e" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 3s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7f9098119ca8614d5cd499412e852e2203597f667ddb305a0977f2ca00b6e53c" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 2s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a07e6f8dfb3d5369cd8ff9d51d033c18e3f97fc96c9264552d7ab6270b8a0078" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Warning FailedCreatePodSandBox 1s kubelet, ubuntu Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6c46db7b766ff80e6e15beb12865961d6d75a31ce9c7b73d049306517598c344" network for pod "nginx": NetworkPlugin cni failed to set up pod "nginx_default" network: failed to set up pod interface "net0" from the device "ens18": failed to open the sriov_numfs of device "ens18": lstat /sys/class/net/ens18/device/sriov_numvfs: no such file or directory
Normal SandboxChanged 0s (x7 over 6s) kubelet, ubuntu Pod sandbox changed, it will be killed and re-created.

[pperiyasamy]

Now I have made following changes to the dpdk-conf network attachment configuration like below:

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: dpdk-conf
spec:
config: '{
"name": "mynet",
"type": "sriov",
"if0": "ens18",
"ifname": "net0",
"dpdk": {
"pci_addr": "0000:00:12.0",
"ifname": "net0",
"vfid": 12,
"kernel_driver":"uio",
"dpdk_driver":"igb_uio",
"dpdk_tool":"/usr/src/dpdk-stable-17.11.4/install/share/dpdk/usertools/dpdk-devbind.py"
},
"deviceinfo": {
"pfname": "ens18",
"pci_addr": "0000:00:12.0",
"vfid": 12
}
}'
EOF

while I try to bring up the POD, it fails with the error "error in invoke Delegate add - "sriov": cannot get a network device with pci address 0000:00:12.0 "lstat /sys/bus/pci/devices/0000:00:12.0/net/: no such file or directory". Actually there is no directory with the name net under the pci address directory.
Should it (getDeviceNameFromPci) be handled differently in case of virtio interfaces ?

sr-iov cni version : dev/k8s-deviceid-model

[zshi-redhat]

@pperiyasamy hi, current sriov-cni is not implemented to support virtio interface yet, the failure above indicates that it cannot get the VF device name according to given VF pci address (even for dpdk config, this is required). Reason for that is the way we get VF name is only applied to SR-IOV VFs, not for virtio interface.

Plus. the code in dev/k8s-deviceid-model branch is intended to work together with another two components or projects (multus and SR-IOV Device Plugin) which can be found under Intel github domain. There are instructions about how to use them in SR-IOV Device Plugin Readme file, but again, virtio case is not yet supported there.

I think your test maybe a valid use case to support: how to schedule the limited host device resource, it might includes SR-IOV VF resource or even virtual resource that are limited(for example, virtio device) in a VM environment.

[rkamudhan]

@pperiyasamy If you create additional interfaces in a VM it should be emulated as a interface inside a VM. So in this case, the virtio should be shown as a interface. Please check whether it showed there or not. if you do ifconfig

@zshi-redhat, we doing the same thing in the kuryr CNI DPDK plugin, Virtio interface is emulated as kernel interfaces inside a VM. We actually do the same implementation of binding it to the DPDK drivers. So @ahalim-intel solution here is generic and it should work similar to Kuryr CNI DPDK plugin.

[zshi-redhat]

@rkamudhan Thanks for the context, I was not aware of how Kuryr CNI DPDK plugin does for virtio.
so you're suggesting that we expand current implementation(sriov device only) to also support virtio interface in dpdk mode?

[ahalimx86]

@pperiyasamy If you create additional interfaces in a VM it should be emulated as a interface inside a VM. So in this case, the virtio should be shown as a interface. Please check whether it showed there or not. if you do ifconfig

I'm not very familiar with virtio and how the interface looks like inside a VM. So, if an sriov nic is added as virtio device on a VM will it appear as regular sriov capable nic with all relevant sysfs entries? If thats the case then there should not be any issues. If not then this plugin won't support it.

[pperiyasamy]

@rkamudhan, @ahalim-intel, @zshi-redhat , Thanks a lot for your response.

Yes, The PCI address net directory for virtio is located at /sys/devices/pci0000:00/<pci_addr>/virtio/net/. Now this virtio interface can be attached to K8S POD container with following configuration.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: dpdk-conf
spec:
config: '{
"name": "mynet",
"type": "sriov",
"if0": "ens18",
"if0name": "net0",
"dpdk": {
"pci_addr": "0000:00:12.0",
"ifname": "ens18",
"vfid": 5,
"kernel_driver":"virtio-pci",
"dpdk_driver":"igb_uio",
"dpdk_tool":"/usr/src/dpdk-stable-17.11.4/install/share/dpdk/usertools/dpdk-devbind.py"
},
"deviceinfo": {
"pfname": "ens18",
"pci_addr": "0000:00:12.0",
"vfid": 5
}
}'
We are also needed to make multus-cni plugin ignoring result.IPs for the cmdAdd operation. The IPs are irrelevant in case of interface is bound to DPDK driver.

Do you have plan to support virtio interface in SR-IOV CNI and its device plugins ?

[zshi-redhat]

@pperiyasamy As Abdul indicated in last comment, I think we don't support virtio in SR-IOV CNI and afaik, there seems no plan of doing so (please correct me if I'm wrong @rkamudhan @ahalim-intel ); we used to run a demo simulating how SR-IOV works with a virtio device plugin, but that project so far is just for demo purpose and doesn't support DPDK.
I feel that this might be a potential request to be able to allocate limited host network devices, particularly if it's in a VM environment, there perhaps are many types of virtual network devices. however, I'm not sure if we shall have all the virtual network devices supported in SR-IOV case.