Error "error adding container to network "k8s-pod-network": Unauthorized".

[HalaharviPedda]

Pods are in "ContainerCreating" state due to authorization problem.
When API server token/certificate get rotated, calico is trying to authenticate using current token, which is invalid as API server token was rotated. Due to this calico is failing to authenticate with API server which results in failing to add network to POD.

Expected Behavior

pods status should be "Running"

Current Behavior

pods report error:
(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5a8497c42b0e1b6aa52eb0f0a8dfd4477db0a615af8e86edf7a6a2cef0b4ce94": plugin type="multus" name="multus-cni-network" failed (add): [kube-system/metrics-server-7968cd6ff4-t6sh2:k8s-pod-network]: error adding container to network "k8s-pod-network": Unauthorized

Possible Solution

Calico-node pod should be somehow check weather API server tokens is rotated, if yes, calico should request token immediately .

Steps to Reproduce (for bugs)

1. Times shift on both control plane and worker nodes and rotate the certificates of API server.
2. We can see few pods are in ContainerCreating, as calico failed to authenticate with API server and to add the network to POD.

Context

Your Environment

• Calico version : v3.24.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): 1.25

[Aleksey-Yermolenko]

This error can also occur when time on nodes is synchronized. There is a suspicion that this happens when OS's maximum number of open files or filesystem-watchers is exceeded (f.ex. fs.inotify.max_user_instances).

Environment:

• Calico v3.24.5 (installed via manifest, with direct connection to ETCD);
• kubernetes v1.26
• containerd v1.6.15

CNI debug logs:
logs.txt

[chenjie901]

i have the same issue about 24h later after install calico
kubernetes 1.26.0
Calico v3.25.0

LAST SEEN              TYPE      REASON                   OBJECT                                   MESSAGE
43m (x9 over 3h47m)    Normal    Sync                     Ingress/nginx-demo                       Scheduled for sync
38m (x2 over 39m)      Normal    Sync                     Ingress/kubernetes-dashboard             Scheduled for sync
10m                    Normal    Killing                  Pod/nginx-deployment-774f96d4d9-9pvjf    Stopping container nginx
10m                    Normal    Killing                  Pod/nginx-deployment-774f96d4d9-5lkst    Stopping container nginx
10m                    Normal    Scheduled                Pod/nginx-deployment-774f96d4d9-p5zv5    Successfully assigned demo/nginx-deployment-774f96d4d9-p5zv5 to k8s-node1
10m                    Normal    Scheduled                Pod/nginx-deployment-774f96d4d9-c7zqx    Successfully assigned demo/nginx-deployment-774f96d4d9-c7zqx to k8s-node1
10m                    Warning   FailedCreatePodSandBox   Pod/nginx-deployment-774f96d4d9-c7zqx    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bfa542ac1f9042bd44bde69b24db846759d8845e46dad23d7c9d4dafd04df12f": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
10m                    Warning   FailedCreatePodSandBox   Pod/nginx-deployment-774f96d4d9-p5zv5    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6f293a19995b17f3dc29eefd9215b8ca02061a876129af7b53e5afca67e27472": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
10m                    Normal    SuccessfulCreate         ReplicaSet/nginx-deployment-774f96d4d9   Created pod: nginx-deployment-774f96d4d9-c7zqx
10m                    Normal    SuccessfulCreate         ReplicaSet/nginx-deployment-774f96d4d9   Created pod: nginx-deployment-774f96d4d9-p5zv5
10m                    Normal    ScalingReplicaSet        Deployment/nginx-deployment              Scaled up replica set nginx-deployment-774f96d4d9 to 2
4m55s (x26 over 10m)   Normal    SandboxChanged           Pod/nginx-deployment-774f96d4d9-c7zqx    Pod sandbox changed, it will be killed and re-created.
22s (x50 over 10m)     Warning   FailedKillPod            Pod/nginx-deployment-774f96d4d9-9pvjf    error killing pod: failed to "KillPodSandbox" for "f2cdae80-22ac-4010-8168-d770d091baa5" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"603c27af3c6a74f21eb2a0bc84bee145ac1a7c4b669285a0823b622296d96c44\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
12s (x50 over 10m)     Warning   FailedKillPod            Pod/nginx-deployment-774f96d4d9-5lkst    error killing pod: failed to "KillPodSandbox" for "d50c8a61-300c-49a2-8f83-c3ee5250afbc" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"4113a8e16035d2e443d488e4aa6f18ff88a8c915d6ff3da2d944c903d5635a77\": plugin type=\"calico\" failed (delete): error getting ClusterInformation: connection is unauthorized: Unauthorized"
0s (x49 over 10m)      Normal    SandboxChanged           Pod/nginx-deployment-774f96d4d9-p5zv5    Pod sandbox changed, it will be killed and re-created.

I reinstall calico，it works again