cannot set "host-device" interface name to "net1": validateIfName: interface name net1 already exists #1089
Comments
|
[root@kubemaster log]# tail -n 200 multus.log |
|
[root@kubemaster ~]# kubectl get pods |
|
[root@kubemaster ~]# ip link show; ip -all netns exec ip link show netns: cni-ff6ab8ee-65a5-b106-f6ea-1364039d5e81 netns: cni-0b4588e1-0f51-5e79-b2ba-d2c244ee9943 netns: cni-71351945-f2ed-da37-3ee1-fa26dd374bef |
|
[root@kubemaster tmp]# kubectl describe pods samplepod-left Normal Scheduled 14s default-scheduler Successfully assigned default/samplepod-left to kubemaster
|
|
I did a fresh install and i can reproduce the issue: If I use the below nad and pod the first time it works ok and adds an additional interface with the host device as the second interface. [root@kubemaster tmp]# cat nad.yaml
Normal Scheduled 3m33s default-scheduler Successfully assigned default/samplepod-left to kubemaster |
|
I removed the existing multus deployment, cleaned up the /etc/cni/net.d/ and /opt/cni/bin/multus I switched to an older branch git checkout release-3.7 I edited the deployment files to add instead of the latest and actual version [root@kubemaster tmp]# git diff
@@ -238,7 +238,7 @@ spec:
@@ -304,7 +304,7 @@ spec:
After multiple tests it seems that it works ok, after restarts @s1061123 @dougbtv seems that there is an issue with latest version of the multus |
|
I have the same issues on a k3s cluster that I am testing with multus v4.0.1. My generated multus config is at: /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conflist It seems like the cni is loading the configuration that already has multus configuration when it is on multus-conf-file=auto. And it does recursively. There's probably some race condition somewhere in the code. Here's my multus config in the beginning (the correct one): {
"cniVersion": "1.0.0",
"name": "multus-cni-network",
"plugins": [ {
"type": "multus",
"capabilities": {"portMappings":true},
"logLevel": "verbose",
"kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
"delegates": [
{"cniVersion":"1.0.0","name":"cbr0","plugins":[{"delegate":{"forceAddress":true,"hairpinMode":true,"isDefaultGateway":true},"type":"flannel"},{"capabilities":{"portMappings":true},"type":"portmap"}]}
]
}]
}then after a few reboot (wrong one): {
"cniVersion": "1.0.0",
"name": "multus-cni-network",
"plugins": [
{
"capabilities": {
"portMappings": true
},
"delegates": [
{
"cniVersion": "1.0.0",
"name": "multus-cni-network",
"plugins": [
{
"capabilities": {
"portMappings": true
},
"delegates": [
{
"cniVersion": "1.0.0",
"name": "multus-cni-network",
"plugins": [
{
"capabilities": {
"portMappings": true
},
"delegates": [
{
"cniVersion": "1.0.0",
"name": "cbr0",
"plugins": [
{
"delegate": {
"forceAddress": true,
"hairpinMode": true,
"isDefaultGateway": true
},
"type": "flannel"
},
{
"capabilities": {
"portMappings": true
},
"type": "portmap"
}
]
}
],
"kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
"logLevel": "verbose",
"type": "multus"
}
]
}
],
"kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
"logLevel": "verbose",
"type": "multus"
}
]
}
],
"kubeconfig": "/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig",
"logLevel": "verbose",
"type": "multus"
}
]
}So my workaround is to delete the multus config: rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conflist and let it regenerate. I also created a simple systemd startup script to delete that file, and so far, it has been working, but it may have other implications that I didn't know of. Of course, I would like to see this fixed properly, not by my hack. Hope this helps. |
|
I found this issue after discovering the same nested delegate misconfiguration and wanted to post here as well. I fixed this issue by adding an initContainer that deletes the multus cni configuration on startup. My patch command looks like: @handoyo-lyrid thanks for posting your multus configuration details - I was able to find this issue as a result.
EDIT 2: Symlinking the k3s cni configuration and binary locations to the ones multus expects prior to running multus seems to have fixed all of my issues. I wrote a DaemonSet to do this for me - see here. I also updated to v4.0.2 of the thin plugin (no arm64 image for the thick plugin) and updated to CNI version 1.0.0. |
|
Any updates on this issue? Or any workaround anyone might has discovered? I have run into this issue every noe and then. Any help will be very appreciated!! Thanks!! |
|
Same here with rk2 installed multus |
|
Same here with RKE2 😢 Do you plan to fix it ? Maybe just put workaround by default on multus daemonset (Still ugly but it works !) Thank you !! |
You can revert to rke2 v1.26.6 it is working. Next version should also get fixed: |
|
I'm running multus v4.0.2 now (thin plugin) with CNI version 1.0.0 on k3s and have fixed most of my issues by deploying a DaemonSet prior to multus that symlinks This also means a custom flannel deployment is not required and the standard multus install can be used without modifications 🎉. EDIT: formatting |
|
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days. |
|
This is a serious issue in the current release and should not be auto closed :-( |
What happend:
Warning FailedCreatePodSandBox 20s (x16 over 35s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ee66bc0cfe368c3512d5d0504cf9d625d465cddf9d5d290a61bceb785f87fc59": plugin type="multus" name="multus-cni-network" failed (add): [default/teravm-testmodule-client-1-0/5f30eb60-e2e9-4545-be53-1e02095d4e92:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net1": validateIfName: interface name net1 already exists
What you expected to happen:
pod to be created with two interfaces, the second one with the host-device CNI plugin
How to reproduce it (as minimally and precisely as possible):
After i created the nad and created the pod with that nad the first time it worked. After a cluster restart, I tried one more time to deploy same pod and I got the above error
[root@kubemaster ~]# cat multus-cni/tmp/nad.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: sriovnetens1f0
spec:
config: '{
"cniVersion": "0.3.0",
"type": "host-device",
"device": "ens1f0",
"ipam": {
"type": "static",
"addresses": [
{
"address": "192.168.1.23/24"
}
]
}
}'
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: sriovnetens1f1
spec:
config: '{
"cniVersion": "0.3.0",
"type": "host-device",
"device": "ens1f1",
"ipam": {
"type": "static",
"addresses": [
{
"address": "192.168.1.24/24"
}
]
}
}'
[root@kubemaster ~]# cat multus-cni/tmp/ex.yaml
apiVersion: v1
kind: Pod
metadata:
name: samplepod-left
annotations:
k8s.v1.cni.cncf.io/networks: sriovnetens1f0@net2
spec:
containers:
command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: nicolaka/netshoot
Anything else we need to know?:
Kubernetes 1.27.1 with Flannel latest version
Environment:
Multus version
image path and image ID (from 'docker images')
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/deployments/multus-daemonset.yml
ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
kube-multus:
Container ID: containerd://d0d567c90ede4dfc12410e6edc8b62ada1a2474ce6bcab936dd50a0647da0140
Image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot
Image ID: ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:95fc9bceac23de1b643da5f4d0fc0a8c5166fa9c1282cac4cb3cbc37b2e5924d
Kubernetes version (use
kubectl version):kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:42Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Primary CNI for Kubernetes cluster:
OS (e.g. from /etc/os-release):
[root@kubemaster ~]# cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.1 (Lime Lynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.1 (Lime Lynx)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
ls /etc/cni/net.d/
00-multus.conf 10-flannel.conflist multus.d
ls /etc/cni/multus/net.d
ls: cannot access '/etc/cni/multus/net.d': No such file or directory
[root@kubemaster bin]# ls
bandwidth bridge dhcp dummy firewall flannel host-device host-local ipvlan loopback macvlan multus portmap ptp sbr static tuning vlan vrf
[root@kubemaster bin]# ./multus version
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
kubectl get net-attach-def -o yaml)kubectl get net-attach-def -o yaml
apiVersion: v1
items:
kind: NetworkAttachmentDefinition
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{},"name":"sriovnetens1f0","namespace":"default"},"spec":{"config":"{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f0", "ipam": { "type": "static", "addresses": [ { "address": "192.168.1.23/24" } ] } }"}}
creationTimestamp: "2023-05-08T17:13:07Z"
generation: 1
name: sriovnetens1f0
namespace: default
resourceVersion: "2582"
uid: 2b8500a2-1599-42d8-bfb8-8ffe7d7a3c31
spec:
config: '{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f0", "ipam":
{ "type": "static", "addresses": [ { "address": "192.168.1.23/24" } ] } }'
kind: NetworkAttachmentDefinition
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{},"name":"sriovnetens1f1","namespace":"default"},"spec":{"config":"{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f1", "ipam": { "type": "static", "addresses": [ { "address": "192.168.1.24/24" } ] } }"}}
creationTimestamp: "2023-05-08T17:13:07Z"
generation: 1
name: sriovnetens1f1
namespace: default
resourceVersion: "2583"
uid: ef32ee6e-9e41-4f2f-987e-7595c8b14d82
spec:
config: '{ "cniVersion": "0.3.0", "type": "host-device", "device": "ens1f1", "ipam":
{ "type": "static", "addresses": [ { "address": "192.168.1.24/24" } ] } }'
kind: List
metadata:
resourceVersion: ""
kubectl get pod <podname> -o yaml)kubectl get pod samplepod-left -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "cbr0",
"interface": "eth0",
"ips": [
"10.244.0.59"
],
"mac": "02:01:2e:21:da:e3",
"default": true,
"dns": {},
"gateway": [
"10.244.0.1"
]
},{
"name": "default/sriovnetens1f0",
"interface": "net2",
"ips": [
"192.168.1.23"
],
"mac": "24:1c:04:f3:d7:fd",
"dns": {}
}]
k8s.v1.cni.cncf.io/networks: sriovnetens1f0@net2
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"sriovnetens1f0@net2"},"name":"samplepod-left","namespace":"default"},"spec":{"containers":[{"command":["/bin/bash","-c","trap : TERM INT; sleep infinity \u0026 wait"],"image":"nicolaka/netshoot","name":"samplepod-left"}]}}
creationTimestamp: "2023-05-08T19:26:27Z"
name: samplepod-left
namespace: default
resourceVersion: "19589"
uid: fade95aa-513f-4f39-860c-b3e8647c3331
spec:
containers:
image: nicolaka/netshoot
imagePullPolicy: Always
name: samplepod-left
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
name: kube-api-access-468wp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
nodeName: kubemaster
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
projected:
defaultMode: 420
sources:
expirationSeconds: 3607
path: token
items:
path: ca.crt
name: kube-root-ca.crt
items:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
lastTransitionTime: "2023-05-08T19:26:27Z"
status: "True"
type: Initialized
lastTransitionTime: "2023-05-08T19:26:27Z"
message: 'containers with unready status: [samplepod-left]'
reason: ContainersNotReady
status: "False"
type: Ready
lastTransitionTime: "2023-05-08T19:26:27Z"
message: 'containers with unready status: [samplepod-left]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
lastTransitionTime: "2023-05-08T19:26:27Z"
status: "True"
type: PodScheduled
containerStatuses:
imageID: ""
lastState: {}
name: samplepod-left
ready: false
restartCount: 0
started: false
state:
waiting:
reason: ContainerCreating
hostIP: 10.160.160.168
phase: Pending
qosClass: BestEffort
startTime: "2023-05-08T19:26:27Z"
journalctl -u kubelet -n 100 -f
May 08 20:34:08 kubemaster kubelet[1970]: E0508 20:34:08.035480 1970 kuberuntime_manager.go:1122] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox "8534602124fe984c934d1335860bf3662a5302b2f457e93f98b2d32792e19d6e": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net2": validateIfName: interface name net2 already exists" pod="default/samplepod-left"
May 08 20:34:08 kubemaster kubelet[1970]: E0508 20:34:08.035531 1970 pod_workers.go:1281] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "samplepod-left_default(fade95aa-513f-4f39-860c-b3e8647c3331)" with CreatePodSandboxError: "Failed to create sandbox for pod \"samplepod-left_default(fade95aa-513f-4f39-860c-b3e8647c3331)\": rpc error: code = Unknown desc = failed to setup network for sandbox \"8534602124fe984c934d1335860bf3662a5302b2f457e93f98b2d32792e19d6e\": plugin type=\"multus\" name=\"multus-cni-network\" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network \"sriovnetens1f0\": DelegateAdd: cannot set \"host-device\" interface name to \"net2\": validateIfName: interface name net2 already exists"" pod="default/samplepod-left" podUID=fade95aa-513f-4f39-860c-b3e8647c3331
May 08 20:34:09 kubemaster kubelet[1970]: E0508 20:34:09.036212 1970 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox "1ad2fb3478d4ed997ac19d0812c72aed408ebfd08ceb30b25abbd4191cfef4df": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net2": validateIfName: interface name net2 already exists"
May 08 20:34:09 kubemaster kubelet[1970]: E0508 20:34:09.036478 1970 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox "1ad2fb3478d4ed997ac19d0812c72aed408ebfd08ceb30b25abbd4191cfef4df": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net2": validateIfName: interface name net2 already exists" pod="default/samplepod-left"
May 08 20:34:09 kubemaster kubelet[1970]: E0508 20:34:09.036502 1970 kuberuntime_manager.go:1122] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox "1ad2fb3478d4ed997ac19d0812c72aed408ebfd08ceb30b25abbd4191cfef4df": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network "sriovnetens1f0": DelegateAdd: cannot set "host-device" interface name to "net2": validateIfName: interface name net2 already exists" pod="default/samplepod-left"
May 08 20:34:09 kubemaster kubelet[1970]: E0508 20:34:09.036563 1970 pod_workers.go:1281] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "samplepod-left_default(fade95aa-513f-4f39-860c-b3e8647c3331)" with CreatePodSandboxError: "Failed to create sandbox for pod \"samplepod-left_default(fade95aa-513f-4f39-860c-b3e8647c3331)\": rpc error: code = Unknown desc = failed to setup network for sandbox \"1ad2fb3478d4ed997ac19d0812c72aed408ebfd08ceb30b25abbd4191cfef4df\": plugin type=\"multus\" name=\"multus-cni-network\" failed (add): [default/samplepod-left/fade95aa-513f-4f39-860c-b3e8647c3331:sriovnetens1f0]: error adding container to network \"sriovnetens1f0\": DelegateAdd: cannot set \"host-device\" interface name to \"net2\": validateIfName: interface name net2 already exists"" pod="default/samplepod-left" podUID=fade95aa-513f-4f39-860c-b3e8647c3331
^C
The text was updated successfully, but these errors were encountered: