Agent status become Not_Ready after reboot

[ivyxjc]

Environmental Info:
K3s Version:
v1.21.4+k3s1 (3e250fd)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:
amd64
Linux gamma 5.4.0-81-generic #91-Ubuntu

Cluster Configuration:

3 servers: 1 master 2 agents

Describe the bug:

Steps To Reproduce:

• Installed K3s (use ansible to install k3s-ansible, no more custom args)
• shutdown one agent
• restart the agent

There is another way to re-produce the issue

• Installed K3s (use ansible to install k3s-ansible, no more custom args)
• restart service k3s in master

Expected behavior:

The shutdown agent should join the cluster automatically, and status should be Ready.

Actual behavior:

The shutdown agent should join the cluster automatically, but status is Not Ready.

Additional context / logs:

Logs

Describe the node or systemctl status k3s,
I can see a lot of logs like

"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Not worked Solution

1. Restart service k3s-node in agent does not work.
2. Restart service k3s in master does not work.

Solution

I get one solution to solve the problem. I find out that the folder /etc/cni/net.d is empty. So I copy /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist to /etc/cni/net.d/ and systemctl restart k3s-node. It works and agent comes to status Ready

Similar issues

kubeadm-issue-1031

Backporting

• Needs backporting to older releases

[brandond]

Is your system for some reason deleting the /etc/cni directory when the node is restarted? K3s does not expect things in /etc/ to be deleted out from under it when restarting.

[ivyxjc]

Is your system for some reason deleting the /etc/cni directory when the node is restarted? K3s does not expect things in /etc/ to be deleted out from under it when restarting.

From my personal experience, there is no file in /etc/cni/net.d, all config file is automatically placed under /var/lib/rancher/k3s/agent/etc/cni/net.d/ after installing k3s. And it works correctly. But it looks like that k3s refer to directory /etc/cni/net.d other than /var/lib/rancher/k3s/agent/etc/cni/net.d/ when I restart the agent.

[brandond]

Are you initially starting it with a different working directory or configuration than is used when you restart it? This isn't an issue I've seen anyone run into before so I'm wondering what about your environment or configuration is unique.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[lidh15]

The same problem encountered. I got the same problem after started k3s by running 'install.sh' without error.
I checked /etc and /etc/cni not found. I copied that one under /var/lib/rancher/k3s/agent/etc and restarted k3s but it didn't work.