Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task failed, error was: runtime: running task on "IP" #3355

Open
Linutux opened this issue Aug 18, 2024 · 7 comments
Open

Task failed, error was: runtime: running task on "IP" #3355

Linutux opened this issue Aug 18, 2024 · 7 comments
Labels
sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. triage/needs-information Indicates an issue needs more information in order to work on it. triage/not-reproducible Indicates an issue can not be reproduced as described. triage/support Indicates an issue that is a support question.

Comments

@Linutux
Copy link

Linutux commented Aug 18, 2024

What happened?

I tried installing a cluster on Hetzner Cloud using the following kubeone.yaml:

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: '1.30.3'
cloudProvider:
  hetzner: {}
  external: true

The kubeadm setup on each node is successfull, but the load balancer in the Hetzner Cloud console shows it as unhealthy and the following error is shown during the install:

INFO[21:03:16 UTC] Initializing Kubernetes on leader...         
INFO[21:03:16 UTC] Running kubeadm...                            node=138.201.xxx.xxx
WARN[21:04:27 UTC] Task failed, error was: runtime: running task on "138.201.xxx.xxx"
ssh: initializing kubeadm cluster
ssh: exec
ssh: popen
Process exited with status 1
set -xeuo pipefail
export "PATH=$PATH:/sbin:/usr/local/bin:/opt/bin"
if [[ -f /etc/kubernetes/admin.conf ]]; then
	sudo kubeadm --v=6 token create r7e3u0.xxxxxxxxxxxxx --ttl 1h0m0s
	exit 0;
fi

sudo kubeadm --v=6 init --config=./kubeone/cfg/master_0.yaml
sudo find /etc/kubernetes/pki/ -name *.crt -exec chmod 600 {} \;

stderr: + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
+ [[ -f /etc/kubernetes/admin.conf ]]
+ sudo kubeadm --v=6 token create r7e3u0.xxxxxxxxxxxxx --ttl 1h0m0s
I0818 21:03:17.415196    4659 token.go:119] [token] validating mixed arguments
I0818 21:03:17.415295    4659 token.go:128] [token] getting Clientsets from kubeconfig file
I0818 21:03:17.415331    4659 cmdutil.go:94] Using kubeconfig file: /etc/kubernetes/admin.conf
I0818 21:03:17.416680    4659 loader.go:395] Config loaded from file:  /etc/kubernetes/admin.conf
I0818 21:03:17.420203    4659 token.go:243] [token] loading configurations
I0818 21:03:17.422548    4659 initconfiguration.go:114] skip CRI socket detection, fill with the default CRI socket unix:///var/run/containerd/containerd.sock
I0818 21:03:17.423043    4659 interface.go:432] Looking for default routes with IPv4 addresses
I0818 21:03:17.423071    4659 interface.go:437] Default route transits interface "eth0"
I0818 21:03:17.423322    4659 interface.go:209] Interface eth0 is up
I0818 21:03:17.423466    4659 interface.go:257] Interface "eth0" has 3 addresses :[138.201.xxx.xxx/32 2a01:xxx:xxxx:xxxx::1/64 fe80::xxxx:xxx:xxxx:xxxx/64].
I0818 21:03:17.423514    4659 interface.go:224] Checking addr  138.201.xxx.xxx/32.
I0818 21:03:17.423536    4659 interface.go:231] IP found 138.201.xxx.xxx
I0818 21:03:17.423568    4659 interface.go:263] Found valid IPv4 address 138.201.xxx.xxx for interface "eth0".
I0818 21:03:17.423585    4659 interface.go:443] Found active IP 138.201.xxx.xxx 
I0818 21:03:17.423672    4659 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I0818 21:03:17.431106    4659 token.go:252] [token] creating token
I0818 21:03:17.433511    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 2 milliseconds
I0818 21:03:18.435089    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 1 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:18.437755    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 2 milliseconds
I0818 21:03:19.438239    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 2 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:19.439914    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:20.440347    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 3 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:20.442067    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:21.443053    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 4 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:21.444910    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:22.445270    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 5 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:22.446744    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:23.447118    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 6 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:23.448664    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:24.450726    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 7 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:24.452938    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:25.453815    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 8 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:25.455909    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 1 milliseconds
I0818 21:03:26.457190    4659 with_retry.go:234] Got a Retry-After 1s response for attempt 9 to https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s
I0818 21:03:26.459392    4659 round_trippers.go:553] GET https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-r7e3u0?timeout=10s  in 2 milliseconds
I0818 21:03:27.462546    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:27.962184    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:28.463141    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:03:28.962443    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:29.462996    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:29.962109    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:30.462560    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:30.962617    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:31.462486    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:31.963281    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:32.461849    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:32.962566    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:33.462522    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:33.963167    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:34.462287    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:34.966430    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 4 milliseconds
I0818 21:03:35.463113    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:35.962832    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:36.462908    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:36.965022    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:03:37.462486    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:37.963105    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:38.462030    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:38.962407    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:39.462918    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:39.962317    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:40.462886    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:40.961766    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:41.461825    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:41.961863    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:42.462099    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:42.963174    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:03:43.463044    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:43.962985    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:44.462344    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:44.963924    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:45.462199    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:45.962628    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:46.462017    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:46.962108    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:47.462492    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:47.963275    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:48.463256    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:48.961446    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:49.462837    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:49.962172    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:50.462460    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:50.963069    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:51.462143    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:51.963452    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:52.462578    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:52.962586    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:53.463429    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:03:53.962381    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:54.462912    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:54.962353    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:55.462428    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:55.962348    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:56.462738    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:56.963173    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:57.462981    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:57.962774    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:58.463028    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:03:58.963161    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:59.462899    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:03:59.964230    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:00.462419    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:00.963659    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:01.461889    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:01.961941    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:02.462543    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:02.963108    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:03.462354    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:03.962525    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:04.463083    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:04.962095    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:05.463062    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:05.962004    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:06.462803    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:06.963047    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:07.463244    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:07.966491    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:08.461877    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:08.962056    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:09.461984    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:09.962720    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:10.462321    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:10.962396    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:11.463820    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:11.962110    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:12.462166    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:12.961983    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:13.462920    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:13.962560    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:14.462987    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:14.962107    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:15.462432    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:15.962631    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:16.462692    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:16.963427    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:17.463279    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:17.962522    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:18.462579    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:18.963458    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:19.463400    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:19.963463    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:20.465080    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:20.962549    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:21.462157    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:21.962595    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:22.463014    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:22.961996    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:23.462005    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:23.962920    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:24.462863    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 2 milliseconds
I0818 21:04:24.962384    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:25.462635    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:25.961677    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:26.462068    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:26.962051    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
I0818 21:04:27.462188    4659 round_trippers.go:553] POST https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
Post "https://49.12.xxx.xxx:6443/api/v1/namespaces/kube-system/secrets?timeout=10s": EOF
unable to create Secret
k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient.CreateOrUpdateSecret.func1
	k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient/idempotency.go:149
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	k8s.io/apimachinery/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	k8s.io/apimachinery/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout
	k8s.io/apimachinery/pkg/util/wait/poll.go:48
k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient.CreateOrUpdateSecret
	k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient/idempotency.go:143
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.UpdateOrCreateTokens.func1
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:59
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
	k8s.io/apimachinery/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	k8s.io/apimachinery/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout
	k8s.io/apimachinery/pkg/util/wait/poll.go:48
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.UpdateOrCreateTokens
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:54
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.CreateNewTokens
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:37
k8s.io/kubernetes/cmd/kubeadm/app/cmd.RunCreateToken
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/token.go:253
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdToken.func1
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/token.go:135
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
	k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:52
main.main
	k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	runtime/proc.go:271
runtime.goexit
	runtime/asm_amd64.s:1695
failed to create or update bootstrap token with name bootstrap-token-r7e3u0
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.UpdateOrCreateTokens.func1
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:60
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
	k8s.io/apimachinery/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	k8s.io/apimachinery/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout
	k8s.io/apimachinery/pkg/util/wait/poll.go:48
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.UpdateOrCreateTokens
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:54
k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node.CreateNewTokens
	k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/node/token.go:37
k8s.io/kubernetes/cmd/kubeadm/app/cmd.RunCreateToken
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/token.go:253
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdToken.func1
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/token.go:135
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
	k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:52
main.main
	k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	runtime/proc.go:271
runtime.goexit
	runtime/asm_amd64.s:1695

Expected behavior

A cluster will be installed.

How to reproduce the issue?

What KubeOne version are you using?

$ kubeone version
{
  "kubeone": {
    "major": "1",
    "minor": "8",
    "gitVersion": "1.8.2",
    "gitCommit": "667a5da2bc1eab61b3183cdfcfc192deef49f985",
    "gitTreeState": "",
    "buildDate": "2024-08-08T15:11:37Z",
    "goVersion": "go1.22.3",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "59",
    "gitVersion": "v1.59.3",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

Provide your KubeOneCluster manifest here (if applicable)

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: '1.30.3'
cloudProvider:
  hetzner: {}
  external: true

What cloud provider are you running on?

Hetzner

What operating system are you running in your cluster?

Ubuntu 22.04

Additional information
@Linutux Linutux added kind/bug Categorizes issue or PR as related to a bug. sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. labels Aug 18, 2024
@xmudrii
Copy link
Member

xmudrii commented Aug 27, 2024

@Linutux Can you please collect logs for kube-apiserver and kubelet, and post them in this ticket (I recommend putting them in a GitHub Gist if there's a lot of logs)? Those kubeadm logs don't tell much, but it's likely that there's something wrong with kubelet or kube-apiserver.

You can grab kubelet logs with sudo journalctl -u kubelet.service and kube-apiserver with crictl (e.g. crictl ps -a and then crictl logs <container-id>.

@xmudrii xmudrii added triage/support Indicates an issue that is a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Aug 27, 2024
@aki-luxor
Copy link

aki-luxor commented Sep 10, 2024 •
edited
Loading

I have something similar issue, it looks like

[5.223.45.173] I0910 23:02:56.582090    3883 kubelet.go:68] Stopping the kubelet
[5.223.45.173] [control-plane] Creating static Pod manifest for "kube-scheduler"
[5.223.45.173] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[5.223.45.173] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[5.223.45.173] [kubelet-start] Starting the kubelet
[5.223.45.173] I0910 23:02:56.840650    3883 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
[5.223.45.173] I0910 23:02:56.841844    3883 loader.go:395] Config loaded from file:  /etc/kubernetes/admin.conf
[5.223.45.173] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[5.223.45.173] I0910 23:02:56.848219    3883 round_trippers.go:553] GET https://x.x.49.45:6443/healthz?timeout=10s  in 4 milliseconds

On control plane node kubelet logs saysjournalctl -xeu kubelet

░░ The job identifier is 1702.
Sep 10 23:00:46 ubuntu-8gb-sin-1-kubeone-5 kubelet[3458]: E0910 23:00:46.708075    3458 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 10 23:00:46 ubuntu-8gb-sin-1-kubeone-5 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited

For some reason this file /var/lib/kubelet/config.yaml is not getting pushed/written to the control plane node

Fixed:

After investigating, I found that I needed to set the IP address of first control plane node in apiEndpoint.host. Once I did this, everything worked as expected. For context, my setup includes 3 control plane nodes and 12 worker nodes.

@Linutux
Copy link
Author

Linutux commented Sep 13, 2024 •
edited
Loading

After investigating, I found that I needed to set the IP address of first control plane node in apiEndpoint.host. Once I did this, everything worked as expected. For context, my setup includes 3 control plane nodes and 12 worker nodes.

This did indeed help. After terraform execution I added the IP of the Hetzner Kubernetes API Loadbalancer to the kubeone.yaml and the installation succeeded:

apiEndpoint:
  host: xxx.xxx.xx.xxx:6443

@kron4eg
Copy link
Member

kron4eg commented Sep 13, 2024

@aki-luxor @Linutux do you have those lines in your terraform?

kubeone/examples/terraform/hetzner/output.tf

Lines 17 to 24 in e50a6b1

output "kubeone_api" {
description = "kube-apiserver LB endpoint"
value = {
endpoint = local.kubeapi_endpoint
apiserver_alternative_names = var.apiserver_alternative_names
}
}

This is the automation to get the apiEndpoint.host from terraform. Is it broken?!

@Linutux
Copy link
Author

Linutux commented Sep 13, 2024

@aki-luxor @Linutux do you have those lines in your terraform?

Yes. My file looks exactly like to one in the repo, I just copied it some week ago. The only terraform file I have modified is the terraform.tfvars.

@kron4eg
Copy link
Member

kron4eg commented Sep 13, 2024

well. I can not reproduce the issue.

my kubeone config

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster

versions:
  kubernetes: 1.29.3

cloudProvider:
  external: true

my terraform state after I've created it

$ tofu state show hcloud_load_balancer.load_balancer[0]
# hcloud_load_balancer.load_balancer[0]:
resource "hcloud_load_balancer" "load_balancer" {
    delete_protection  = false
    id                 = "2055670"
    ipv4               = "167.233.8.33"
    ipv6               = "2a01:4f8:1c1f:66cc::1"
    labels             = {
        "kubeone_cluster_name" = "artioms"
        "role"                 = "lb"
    }
    load_balancer_type = "lb11"
    location           = "nbg1"
    name               = "artioms-lb"
    network_zone       = "eu-central"

    algorithm {
        type = "round_robin"
    }
}

my terraform outputs

$ tofu output -json | jq .kubeone_api.value
{
  "apiserver_alternative_names": [],
  "endpoint": "167.233.8.33"
}

and then I have it merged at runtime into kubeone config (in memory)

$ kubeone config dump -t tf.json -m kubeone.yaml | yq .apiEndpoint
host: 167.233.8.33
port: 6443
---
null

kubeone config dump command will reveal the kubeone config after all the defaulting and merging with terraform integration (if given).

So that all together worked.

@kron4eg kron4eg added triage/needs-information Indicates an issue needs more information in order to work on it. triage/not-reproducible Indicates an issue can not be reproduced as described. labels Sep 13, 2024
@aki-luxor
Copy link

@aki-luxor @Linutux do you have those lines in your terraform?

kubeone/examples/terraform/hetzner/output.tf

Lines 17 to 24 in e50a6b1

output "kubeone_api" {
description = "kube-apiserver LB endpoint"
value = {
endpoint = local.kubeapi_endpoint
apiserver_alternative_names = var.apiserver_alternative_names
}
}

This is the automation to get the apiEndpoint.host from terraform. Is it broken?!

Sorry for not making this clear earlier, I am not using TF module. I was provisioning cluster manually without TF automation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sig/cluster-management Denotes a PR or issue as being assigned to SIG Cluster Management. triage/needs-information Indicates an issue needs more information in order to work on it. triage/not-reproducible Indicates an issue can not be reproduced as described. triage/support Indicates an issue that is a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kron4eg @Linutux @xmudrii @aki-luxor