Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CrashLoopBackOff / Error on Raspberry Pi 4 #3389

Closed
lauchokyip opened this issue Jun 2, 2021 · 7 comments
Closed

CrashLoopBackOff / Error on Raspberry Pi 4 #3389

lauchokyip opened this issue Jun 2, 2021 · 7 comments

Comments

@lauchokyip
Copy link

This issue is used to track the CrashLoopBackOff error on Raspberry Pi 4 instead of Cent OS
Environmental Info:
K3s Version:

k3s version v1.21.1+k3s1 (75dba57f)
go version go1.16.4

Node(s) CPU architecture, OS, and Version:

Linux kmaster 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux

Cluster Configuration:

1 server, 1 agent (both are Raspberry pi 4)

Describe the bug:

Opening another issue as suggested #1019 (comment) to keep track of the pod CrashLoopBackOff

:~ $ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE
kube-system   helm-install-traefik-crd-krv7p            0/1     Completed          0          26m
kube-system   helm-install-traefik-vn7wv                0/1     Completed          2          26m
kube-system   svclb-traefik-wrrkh                       2/2     Running            2          25m
kube-system   traefik-97b44b794-gwkss                   1/1     Running            1          25m
kube-system   svclb-traefik-q8qhq                       2/2     Running            0          15m
kube-system   coredns-7448499f4d-f7kjn                  0/1     CrashLoopBackOff   5          26m
kube-system   metrics-server-86cbb8457f-b7dnm           0/1     CrashLoopBackOff   5          26m
kube-system   local-path-provisioner-5ff76fc89d-q5tpr   0/1     Error              6          26m

Steps To Reproduce:

  1. After making sure both master node and worker node are ready reboot both raspberry pi
  2. After rebooting,
:~ $ kubectl get nodes
NAME      STATUS     ROLES                  AGE   VERSION
knode0    NotReady   <none>                 15m   v1.21.1+k3s1
kmaster   Ready      control-plane,master   26m   v1.21.1+k3s1

Expected behavior:
Expected both nodes to be ready

Actual behavior:
worker node was never ready

Additional context / logs:
If I execute the kill-all script and reboot the server with k3s server start it starts working again, but this is inconvenient.
@brandond
Copy link
Member

brandond commented Jun 2, 2021 •
edited
Loading

Just seeing that they're crashlooping doesn't give us much to work with. Can you provide kubectl logs and kubectl describe pod output from those two pods?

For the not-ready node, what do the k3s logs show?

@lauchokyip
Copy link
Author

After rebooting, was able to reproduce again

NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE
kube-system   helm-install-traefik-crd-krv7p            0/1     Completed          0          102m
kube-system   helm-install-traefik-vn7wv                0/1     Completed          2          102m
kube-system   svclb-traefik-wrrkh                       2/2     Running            10         100m
kube-system   traefik-97b44b794-gwkss                   1/1     Running            6          100m
kube-system   svclb-traefik-q8qhq                       2/2     Running            2          91m
kube-system   metrics-server-86cbb8457f-b7dnm           0/1     CrashLoopBackOff   16         102m
kube-system   local-path-provisioner-5ff76fc89d-q5tpr   0/1     CrashLoopBackOff   18         102m
kube-system   coredns-7448499f4d-f7kjn                  0/1     CrashLoopBackOff   18         102m

For metrics-server-86cbb8457f-b7dnm

lau@debian:~ $ kubectl logs -n kube-system metrics-server-86cbb8457f-b7dnm
I0602 17:58:04.650011       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
Error: Get https://10.43.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.43.0.1:443: connect: connection refused
Usage:
   [flags]

Flags:
      --alsologtostderr                                         log to standard error as well as files
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authentication-tolerate-lookup-failure                  If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
      --authorization-always-allow-paths strings                A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
      --kubeconfig string                                       The path to the kubeconfig used to connect to the Kubernetes API server and the Kubelets (defaults to in-cluster config)
      --kubelet-certificate-authority string                    Path to the CA to use to validate the Kubelet's serving certificates.
      --kubelet-insecure-tls                                    Do not verify CA of serving certificates presented by Kubelets.  For testing purposes only.
      --kubelet-port int                                        The port to use to connect to Kubelets. (default 10250)
      --kubelet-preferred-address-types strings                 The priority of node address types to use when determining which address to use to connect to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --log_file string                                         If non-empty, use this log file
      --logtostderr                                             log to standard error instead of files (default true)
      --metric-resolution duration                              The resolution at which metrics-server will retain metrics. (default 1m0s)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names strings                     List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      --requestheader-extra-headers-prefix strings              List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers strings                     List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers strings                  List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 443)
      --skip_headers                                            If true, avoid header prefixes in the log messages
      --stderrthreshold severity                                logs at or above this threshold go to stderr
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites strings                               Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
      --tls-min-version string                                  Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
  -v, --v Level                                                 number for the log level verbosity
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging

panic: Get https://10.43.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.43.0.1:443: connect: connection refused

goroutine 1 [running]:
main.main()
	/go/src/github.com/kubernetes-incubator/metrics-server/cmd/metrics-server/metrics-server.go:39 +0x10c

lau@debian:~ $ kubectl describe -n kube-system pods metrics-server-86cbb8457f-b7dnm
Name:                 metrics-server-86cbb8457f-b7dnm
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 kmaster/192.168.0.21
Start Time:           Wed, 02 Jun 2021 12:14:05 -0400
Labels:               k8s-app=metrics-server
                      pod-template-hash=86cbb8457f
Annotations:          <none>
Status:               Running
IP:                   10.42.0.17
IPs:
  IP:           10.42.0.17
Controlled By:  ReplicaSet/metrics-server-86cbb8457f
Containers:
  metrics-server:
    Container ID:   containerd://6b4aca354e1be2fef0e607ddbdc1d53f3fddabe4fe9e12889fd66796cbefb963
    Image:          rancher/metrics-server:v0.3.6
    Image ID:       docker.io/rancher/metrics-server@sha256:b85628b103169d7db52a32a48b46d8942accb7bde3709c0a4888a23d035f9f1e
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 02 Jun 2021 13:57:57 -0400
      Finished:     Wed, 02 Jun 2021 13:58:10 -0400
    Ready:          False
    Restart Count:  18
    Environment:    <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w27w2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-w27w2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               105m                   default-scheduler  Successfully assigned kube-system/metrics-server-86cbb8457f-b7dnm to kmaster
  Normal   Pulling                 105m                   kubelet            Pulling image "rancher/metrics-server:v0.3.6"
  Normal   Pulled                  104m                   kubelet            Successfully pulled image "rancher/metrics-server:v0.3.6" in 20.181006925s
  Normal   Created                 104m                   kubelet            Created container metrics-server
  Normal   Started                 104m                   kubelet            Started container metrics-server
  Normal   SandboxChanged          84m                    kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  82m (x4 over 84m)      kubelet            Container image "rancher/metrics-server:v0.3.6" already present on machine
  Normal   Created                 82m (x4 over 84m)      kubelet            Created container metrics-server
  Normal   Started                 82m (x4 over 84m)      kubelet            Started container metrics-server
  Warning  BackOff                 79m (x20 over 84m)     kubelet            Back-off restarting failed container
  Warning  FailedMount             77m                    kubelet            MountVolume.SetUp failed for volume "kube-api-access-w27w2" : [failed to fetch token: serviceaccounts "metrics-server" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object, failed to sync configmap cache: timed out waiting for the condition]
  Warning  FailedCreatePodSandBox  77m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9131bc6ba17249ed7611ced6bcd8ee5cb3115f4e6d7e2788fd2a4e70f4cc16eb": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  77m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "568e3c8fa55ea1af77488c9e09c74cf690299250612dc2d012a6dc760ba011f1": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          75m (x3 over 77m)      kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  72m (x4 over 74m)      kubelet            Container image "rancher/metrics-server:v0.3.6" already present on machine
  Normal   Created                 72m (x4 over 74m)      kubelet            Created container metrics-server
  Normal   Started                 72m (x4 over 74m)      kubelet            Started container metrics-server
  Warning  BackOff                 70m (x18 over 74m)     kubelet            Back-off restarting failed container
  Warning  FailedMount             68m                    kubelet            MountVolume.SetUp failed for volume "kube-api-access-w27w2" : failed to fetch token: serviceaccounts "metrics-server" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object
  Normal   SandboxChanged          68m                    kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  68m                    kubelet            Container image "rancher/metrics-server:v0.3.6" already present on machine
  Normal   Created                 68m                    kubelet            Created container metrics-server
  Normal   Started                 68m                    kubelet            Started container metrics-server
  Warning  FailedCreatePodSandBox  11m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e8ea00cdd92b4994e795eb9e6a3d2e4a12798f40c3124014c9696e54c1013df9": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  11m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a9ace9c7e119e6c7c9ffcbe2d94ca533d11304cccc17d9916cb771052573dd58": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          10m (x3 over 11m)      kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  10m                    kubelet            Container image "rancher/metrics-server:v0.3.6" already present on machine
  Normal   Created                 10m                    kubelet            Created container metrics-server
  Normal   Started                 10m                    kubelet            Started container metrics-server
  Warning  BackOff                 10m                    kubelet            Back-off restarting failed container
  Warning  FailedCreatePodSandBox  10m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "81757c458230e3be4b84bc643cb345109f21b49d1bdb752e00475c9b0daa0f67": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  10m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0f5ad8cef733b2c01b72db4f7a89f48fc06db19c316ae06d4bc06243f4c961fb": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          7m43s (x3 over 10m)    kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   SandboxChanged          4m59s                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  2m54s (x4 over 4m57s)  kubelet            Container image "rancher/metrics-server:v0.3.6" already present on machine
  Normal   Created                 2m54s (x4 over 4m56s)  kubelet            Created container metrics-server
  Normal   Started                 2m54s (x4 over 4m56s)  kubelet            Started container metrics-server
  Warning  BackOff                 88s (x12 over 4m44s)   kubelet            Back-off restarting failed container

For local-path-provisioner-5ff76fc89d-q5tpr

lau@debian:~ $ kubectl logs -n kube-system local-path-provisioner-5ff76fc89d-q5tpr 
time="2021-06-02T18:00:05Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get https://10.43.0.1:443/version?timeout=32s: dial tcp 10.43.0.1:443: connect: connection refused" 

Name:                 local-path-provisioner-5ff76fc89d-q5tpr
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 kmaster/192.168.0.21
Start Time:           Wed, 02 Jun 2021 12:14:05 -0400
Labels:               app=local-path-provisioner
                      pod-template-hash=5ff76fc89d
Annotations:          <none>
Status:               Running
IP:                   10.42.0.18
IPs:
  IP:           10.42.0.18
Controlled By:  ReplicaSet/local-path-provisioner-5ff76fc89d
Containers:
  local-path-provisioner:
    Container ID:  containerd://7319d0a1c3834f8b2dca72d7d9ac58783aa8a34ccfc39c82fc20bac274303636
    Image:         rancher/local-path-provisioner:v0.0.19
    Image ID:      docker.io/rancher/local-path-provisioner@sha256:9666b1635fec95d4e2251661e135c90678b8f45fd0f8324c55db99c80e2a958c
    Port:          <none>
    Host Port:     <none>
    Command:
      local-path-provisioner
      start
      --config
      /etc/config/config.json
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 02 Jun 2021 14:00:05 -0400
      Finished:     Wed, 02 Jun 2021 14:00:05 -0400
    Ready:          False
    Restart Count:  20
    Environment:
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /etc/config/ from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrsg4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      local-path-config
    Optional:  false
  kube-api-access-xrsg4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               107m                    default-scheduler  Successfully assigned kube-system/local-path-provisioner-5ff76fc89d-q5tpr to kmaster
  Normal   Pulling                 107m                    kubelet            Pulling image "rancher/local-path-provisioner:v0.0.19"
  Normal   Pulled                  107m                    kubelet            Successfully pulled image "rancher/local-path-provisioner:v0.0.19" in 14.544771079s
  Normal   Created                 107m                    kubelet            Created container local-path-provisioner
  Normal   Started                 107m                    kubelet            Started container local-path-provisioner
  Normal   SandboxChanged          87m                     kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  86m (x4 over 87m)       kubelet            Container image "rancher/local-path-provisioner:v0.0.19" already present on machine
  Normal   Created                 86m (x4 over 87m)       kubelet            Created container local-path-provisioner
  Normal   Started                 86m (x4 over 87m)       kubelet            Started container local-path-provisioner
  Warning  BackOff                 82m (x25 over 87m)      kubelet            Back-off restarting failed container
  Warning  FailedMount             80m                     kubelet            MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount             80m                     kubelet            MountVolume.SetUp failed for volume "kube-api-access-xrsg4" : [failed to fetch token: serviceaccounts "local-path-provisioner-service-account" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object, failed to sync configmap cache: timed out waiting for the condition]
  Warning  FailedCreatePodSandBox  80m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f29b3fb130131508ffd544d3a2c4ef56b31098ec0a38103d0d5364fcc8000824": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  80m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f8b02f2a65e73de8e888eee4669a6a25017d25c06ce703c4110ecfcc9951e34e": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          77m (x3 over 80m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Started                 76m (x3 over 77m)       kubelet            Started container local-path-provisioner
  Normal   Pulled                  76m (x4 over 77m)       kubelet            Container image "rancher/local-path-provisioner:v0.0.19" already present on machine
  Normal   Created                 76m (x4 over 77m)       kubelet            Created container local-path-provisioner
  Warning  BackOff                 72m (x22 over 77m)      kubelet            Back-off restarting failed container
  Warning  FailedMount             71m                     kubelet            MountVolume.SetUp failed for volume "kube-api-access-xrsg4" : failed to fetch token: serviceaccounts "local-path-provisioner-service-account" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object
  Normal   SandboxChanged          71m                     kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff                 70m                     kubelet            Back-off restarting failed container
  Normal   Pulled                  70m (x2 over 71m)       kubelet            Container image "rancher/local-path-provisioner:v0.0.19" already present on machine
  Normal   Created                 70m (x2 over 71m)       kubelet            Created container local-path-provisioner
  Normal   Started                 70m (x2 over 71m)       kubelet            Started container local-path-provisioner
  Warning  FailedCreatePodSandBox  14m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b344ef4137b02fbe78e6aad2105a2e0b14495ff173730ae59afe7c8154d54f30": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          13m (x2 over 14m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  13m                     kubelet            Container image "rancher/local-path-provisioner:v0.0.19" already present on machine
  Normal   Created                 13m                     kubelet            Created container local-path-provisioner
  Normal   Started                 13m                     kubelet            Started container local-path-provisioner
  Warning  BackOff                 13m (x2 over 13m)       kubelet            Back-off restarting failed container
  Warning  FailedCreatePodSandBox  12m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5f733db6e7fd368e251e69187d270adaeaf30b1fc620f16794c1e95ee8e84e0": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          12m (x2 over 12m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  12m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ae4a5d1471357c391e44aa6782212167ca6f37ae7161fb1873997a472c799ffb": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          7m45s                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  6m9s (x4 over 7m43s)    kubelet            Container image "rancher/local-path-provisioner:v0.0.19" already present on machine
  Normal   Created                 6m9s (x4 over 7m42s)    kubelet            Created container local-path-provisioner
  Normal   Started                 6m9s (x4 over 7m42s)    kubelet            Started container local-path-provisioner
  Warning  BackOff                 2m44s (x25 over 7m39s)  kubelet            Back-off restarting failed container

For coredns-7448499f4d-f7kjn

lau@debian:~ $ kubectl logs -n kube-system coredns-7448499f4d-f7kjn
plugin/kubernetes: Get "https://10.43.0.1:443/version?timeout=32s": dial tcp 10.43.0.1:443: connect: connection refused

lau@debian:~ $ kubectl describe  -n kube-system pod coredns-7448499f4d-f7kjn
Name:                 coredns-7448499f4d-f7kjn
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 kmaster/192.168.0.21
Start Time:           Wed, 02 Jun 2021 12:14:05 -0400
Labels:               k8s-app=kube-dns
                      pod-template-hash=7448499f4d
Annotations:          <none>
Status:               Running
IP:                   10.42.0.16
IPs:
  IP:           10.42.0.16
Controlled By:  ReplicaSet/coredns-7448499f4d
Containers:
  coredns:
    Container ID:  containerd://2003e2cd3a50aafd845bca6d53c2784bf78354cb639f48c8673cc12730ac0844
    Image:         rancher/coredns-coredns:1.8.3
    Image ID:      docker.io/rancher/coredns-coredns@sha256:cd0f6ef34a9ee6145c0bcbc0a4511a04f81dc2c91582140f0f0250be834b5f27
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 02 Jun 2021 14:00:11 -0400
      Finished:     Wed, 02 Jun 2021 14:00:11 -0400
    Ready:          False
    Restart Count:  20
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xjrzn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-xjrzn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               109m                    default-scheduler  Successfully assigned kube-system/coredns-7448499f4d-f7kjn to kmaster
  Normal   Pulling                 109m                    kubelet            Pulling image "rancher/coredns-coredns:1.8.3"
  Normal   Pulled                  109m                    kubelet            Successfully pulled image "rancher/coredns-coredns:1.8.3" in 20.229897523s
  Normal   Created                 108m                    kubelet            Created container coredns
  Normal   Started                 108m                    kubelet            Started container coredns
  Warning  DNSConfigForming        92m (x18 over 109m)     kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Normal   SandboxChanged          89m                     kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Created                 88m (x2 over 89m)       kubelet            Created container coredns
  Normal   Pulled                  88m (x2 over 89m)       kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Started                 88m (x2 over 89m)       kubelet            Started container coredns
  Warning  BackOff                 88m (x7 over 88m)       kubelet            Back-off restarting failed container
  Warning  DNSConfigForming        84m (x39 over 89m)      kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Warning  FailedMount             81m                     kubelet            MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount             81m                     kubelet            MountVolume.SetUp failed for volume "kube-api-access-xjrzn" : [failed to fetch token: serviceaccounts "coredns" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object, failed to sync configmap cache: timed out waiting for the condition]
  Warning  FailedCreatePodSandBox  81m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b8990d62872cf3315d34dcafafb42d96c053453c8bef8987724ea5c360797836": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  81m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9f5920a4993e5d1d6e47c92696d2404ed6f2c648c39e320ea5ef853fde57d528": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          81m (x3 over 81m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  79m                     kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 79m                     kubelet            Created container coredns
  Normal   Started                 79m                     kubelet            Started container coredns
  Warning  Unhealthy               79m (x5 over 79m)       kubelet            Readiness probe failed: Get "http://10.42.0.14:8181/ready": dial tcp 10.42.0.14:8181: connect: connection refused
  Warning  BackOff                 79m                     kubelet            Back-off restarting failed container
  Warning  DNSConfigForming        74m (x40 over 81m)      kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Warning  FailedMount             72m                     kubelet            MountVolume.SetUp failed for volume "kube-api-access-xjrzn" : failed to fetch token: serviceaccounts "coredns" is forbidden: User "system:node:kmaster" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'kmaster' and this object
  Warning  FailedMount             72m                     kubelet            MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Normal   SandboxChanged          72m                     kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  72m                     kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 72m                     kubelet            Created container coredns
  Normal   Started                 72m                     kubelet            Started container coredns
  Warning  Unhealthy               72m (x15 over 72m)      kubelet            Readiness probe failed: Get "http://10.42.0.5:8181/ready": dial tcp 10.42.0.5:8181: connect: connection refused
  Warning  DNSConfigForming        17m (x49 over 72m)      kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Warning  FailedCreatePodSandBox  15m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b825e249d58e759b1d5001994196bc02d641aa051a5b11005753e462d24b8c9b": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  15m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "faf7f833f7c69b7a900e568b5cd67811814e9b5629d6b7e9d768ca878189fb9b": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  15m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f6d2f05091d34723758aad0e875ea2c43a1f783b63d38f313582ae14ff141235": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          15m (x4 over 16m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  15m                     kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 15m                     kubelet            Created container coredns
  Normal   Started                 15m                     kubelet            Started container coredns
  Warning  Unhealthy               15m                     kubelet            Readiness probe failed: Get "http://10.42.0.9:8181/ready": dial tcp 10.42.0.9:8181: connect: connection refused
  Warning  DNSConfigForming        15m (x9 over 16m)       kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Warning  BackOff                 15m (x3 over 15m)       kubelet            Back-off restarting failed container
  Warning  FailedCreatePodSandBox  14m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "07a726c282f035352597bd8f0a84561f6af74b803977449e488b3d728a1e7439": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  14m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "32e9d6bd2c62d97eae2233d0dd4db69d9c723e5e03491c77870fcddf9c0bdeb6": open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          12m (x3 over 14m)       kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  12m                     kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 12m                     kubelet            Created container coredns
  Normal   Started                 12m                     kubelet            Started container coredns
  Warning  DNSConfigForming        12m (x7 over 14m)       kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1
  Warning  Unhealthy               12m (x3 over 12m)       kubelet            Readiness probe failed: Get "http://10.42.0.12:8181/ready": dial tcp 10.42.0.12:8181: connect: connection refused
  Normal   SandboxChanged          9m20s                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy               9m14s (x2 over 9m16s)   kubelet            Readiness probe failed: Get "http://10.42.0.16:8181/ready": dial tcp 10.42.0.16:8181: connect: connection refused
  Normal   Pulled                  8m50s (x2 over 9m18s)   kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
  Normal   Created                 8m50s (x2 over 9m17s)   kubelet            Created container coredns
  Normal   Started                 8m50s (x2 over 9m17s)   kubelet            Started container coredns
  Warning  BackOff                 8m48s (x5 over 9m13s)   kubelet            Back-off restarting failed container
  Warning  DNSConfigForming        4m18s (x39 over 9m20s)  kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1

@lauchokyip
Copy link
Author

Logs from master node,

Jun 02 14:04:04 kmaster k3s[619]: time="2021-06-02T14:04:04.987213988-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:05 kmaster k3s[619]: W0602 14:04:05.023541     619 handler_proxy.go:102] no RequestInfo found in the context
Jun 02 14:04:05 kmaster k3s[619]: E0602 14:04:05.024210     619 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Jun 02 14:04:05 kmaster k3s[619]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Jun 02 14:04:05 kmaster k3s[619]: I0602 14:04:05.024905     619 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Jun 02 14:04:05 kmaster k3s[619]: time="2021-06-02T14:04:05.717530506-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:05 http: TLS handshake error from [2601:152:4001:aa90::1f]:46004: remote error: tls: bad certificate"
Jun 02 14:04:06 kmaster k3s[619]: I0602 14:04:06.020831     619 scope.go:111] "RemoveContainer" containerID="2003e2cd3a50aafd845bca6d53c2784bf78354cb639f48c8673cc12730ac0844"
Jun 02 14:04:06 kmaster k3s[619]: I0602 14:04:06.023683     619 scope.go:111] "RemoveContainer" containerID="7319d0a1c3834f8b2dca72d7d9ac58783aa8a34ccfc39c82fc20bac274303636"
Jun 02 14:04:06 kmaster k3s[619]: E0602 14:04:06.023845     619 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1"
Jun 02 14:04:06 kmaster k3s[619]: E0602 14:04:06.025226     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-5ff76fc89d-q5tpr_kube-system(92735307-318f-46d0-bf15-5e3ee9dc554b)\"" pod="kube-system/local-path-provisioner-5ff76fc89d-q5tpr" podUID=92735307-318f-46d0-bf15-5e3ee9dc554b
Jun 02 14:04:06 kmaster k3s[619]: E0602 14:04:06.026316     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=coredns pod=coredns-7448499f4d-f7kjn_kube-system(12a89af8-4450-4ae2-92d1-97fa2d32d1c0)\"" pod="kube-system/coredns-7448499f4d-f7kjn" podUID=12a89af8-4450-4ae2-92d1-97fa2d32d1c0
Jun 02 14:04:08 kmaster k3s[619]: W0602 14:04:08.254363     619 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Jun 02 14:04:08 kmaster k3s[619]: E0602 14:04:08.290617     619 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Jun 02 14:04:08 kmaster k3s[619]: W0602 14:04:08.312539     619 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
Jun 02 14:04:09 kmaster k3s[619]: time="2021-06-02T14:04:09.987644307-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 14:04:10 kmaster k3s[619]: time="2021-06-02T14:04:10.012898243-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:10 kmaster k3s[619]: time="2021-06-02T14:04:10.013131611-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:10 kmaster k3s[619]: time="2021-06-02T14:04:10.014630152-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:10 http: TLS handshake error from [2601:152:4001:aa90::1e]:54004: remote error: tls: bad certificate"
Jun 02 14:04:10 kmaster k3s[619]: time="2021-06-02T14:04:10.751329948-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:10 http: TLS handshake error from [2601:152:4001:aa90::1f]:46006: remote error: tls: bad certificate"
Jun 02 14:04:15 kmaster k3s[619]: time="2021-06-02T14:04:15.013851104-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 14:04:15 kmaster k3s[619]: time="2021-06-02T14:04:15.024158760-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:15 kmaster k3s[619]: time="2021-06-02T14:04:15.024239759-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:15 kmaster k3s[619]: time="2021-06-02T14:04:15.024489571-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:15 http: TLS handshake error from [2601:152:4001:aa90::1e]:54010: remote error: tls: bad certificate"
Jun 02 14:04:15 kmaster k3s[619]: I0602 14:04:15.025698     619 scope.go:111] "RemoveContainer" containerID="8d68ce8251cf6ad57ba3f95e770be2b9c4b4c742281bfeb0b77803c273bb5e32"
Jun 02 14:04:15 kmaster k3s[619]: E0602 14:04:15.026441     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-86cbb8457f-b7dnm_kube-system(d7744bae-ace0-4151-86a4-36fe9dfa440d)\"" pod="kube-system/metrics-server-86cbb8457f-b7dnm" podUID=d7744bae-ace0-4151-86a4-36fe9dfa440d
Jun 02 14:04:15 kmaster k3s[619]: time="2021-06-02T14:04:15.768644453-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:15 http: TLS handshake error from [2601:152:4001:aa90::1f]:46008: remote error: tls: bad certificate"
Jun 02 14:04:19 kmaster k3s[619]: I0602 14:04:19.020511     619 scope.go:111] "RemoveContainer" containerID="7319d0a1c3834f8b2dca72d7d9ac58783aa8a34ccfc39c82fc20bac274303636"
Jun 02 14:04:19 kmaster k3s[619]: E0602 14:04:19.025362     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-5ff76fc89d-q5tpr_kube-system(92735307-318f-46d0-bf15-5e3ee9dc554b)\"" pod="kube-system/local-path-provisioner-5ff76fc89d-q5tpr" podUID=92735307-318f-46d0-bf15-5e3ee9dc554b
Jun 02 14:04:20 kmaster k3s[619]: I0602 14:04:20.019917     619 scope.go:111] "RemoveContainer" containerID="2003e2cd3a50aafd845bca6d53c2784bf78354cb639f48c8673cc12730ac0844"
Jun 02 14:04:20 kmaster k3s[619]: E0602 14:04:20.020350     619 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1"
Jun 02 14:04:20 kmaster k3s[619]: E0602 14:04:20.023091     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=coredns pod=coredns-7448499f4d-f7kjn_kube-system(12a89af8-4450-4ae2-92d1-97fa2d32d1c0)\"" pod="kube-system/coredns-7448499f4d-f7kjn" podUID=12a89af8-4450-4ae2-92d1-97fa2d32d1c0
Jun 02 14:04:20 kmaster k3s[619]: time="2021-06-02T14:04:20.025273751-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 14:04:20 kmaster k3s[619]: time="2021-06-02T14:04:20.048397183-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:20 http: TLS handshake error from [2601:152:4001:aa90::1e]:54018: remote error: tls: bad certificate"
Jun 02 14:04:20 kmaster k3s[619]: time="2021-06-02T14:04:20.048652291-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:20 kmaster k3s[619]: time="2021-06-02T14:04:20.048820049-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:20 kmaster k3s[619]: time="2021-06-02T14:04:20.786748619-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:20 http: TLS handshake error from [2601:152:4001:aa90::1f]:46010: remote error: tls: bad certificate"
Jun 02 14:04:25 kmaster k3s[619]: time="2021-06-02T14:04:25.049688328-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 14:04:25 kmaster k3s[619]: time="2021-06-02T14:04:25.069687160-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:25 kmaster k3s[619]: time="2021-06-02T14:04:25.069844547-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:25 kmaster k3s[619]: time="2021-06-02T14:04:25.069754011-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:25 http: TLS handshake error from [2601:152:4001:aa90::1e]:54028: remote error: tls: bad certificate"
Jun 02 14:04:25 kmaster k3s[619]: time="2021-06-02T14:04:25.803829381-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:25 http: TLS handshake error from [2601:152:4001:aa90::1f]:46012: remote error: tls: bad certificate"
Jun 02 14:04:27 kmaster k3s[619]: I0602 14:04:27.020444     619 scope.go:111] "RemoveContainer" containerID="8d68ce8251cf6ad57ba3f95e770be2b9c4b4c742281bfeb0b77803c273bb5e32"
Jun 02 14:04:27 kmaster k3s[619]: E0602 14:04:27.021798     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-86cbb8457f-b7dnm_kube-system(d7744bae-ace0-4151-86a4-36fe9dfa440d)\"" pod="kube-system/metrics-server-86cbb8457f-b7dnm" podUID=d7744bae-ace0-4151-86a4-36fe9dfa440d
Jun 02 14:04:30 kmaster k3s[619]: time="2021-06-02T14:04:30.070556046-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 14:04:30 kmaster k3s[619]: time="2021-06-02T14:04:30.093155367-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:30 http: TLS handshake error from [2601:152:4001:aa90::1e]:54036: remote error: tls: bad certificate"
Jun 02 14:04:30 kmaster k3s[619]: time="2021-06-02T14:04:30.093245347-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:30 kmaster k3s[619]: time="2021-06-02T14:04:30.093480437-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 14:04:30 kmaster k3s[619]: time="2021-06-02T14:04:30.829030797-04:00" level=info msg="Cluster-Http-Server 2021/06/02 14:04:30 http: TLS handshake error from [2601:152:4001:aa90::1f]:46014: remote error: tls: bad certificate"
Jun 02 14:04:31 kmaster k3s[619]: I0602 14:04:31.024980     619 scope.go:111] "RemoveContainer" containerID="7319d0a1c3834f8b2dca72d7d9ac58783aa8a34ccfc39c82fc20bac274303636"
Jun 02 14:04:31 kmaster k3s[619]: E0602 14:04:31.026548     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-5ff76fc89d-q5tpr_kube-system(92735307-318f-46d0-bf15-5e3ee9dc554b)\"" pod="kube-system/local-path-provisioner-5ff76fc89d-q5tpr" podUID=92735307-318f-46d0-bf15-5e3ee9dc554b
Jun 02 14:04:32 kmaster k3s[619]: I0602 14:04:32.019662     619 scope.go:111] "RemoveContainer" containerID="2003e2cd3a50aafd845bca6d53c2784bf78354cb639f48c8673cc12730ac0844"
Jun 02 14:04:32 kmaster k3s[619]: E0602 14:04:32.019904     619 dns.go:136] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 75.75.75.75 75.75.76.76 2001:558:feed::1"
Jun 02 14:04:32 kmaster k3s[619]: E0602 14:04:32.021079     619 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=coredns pod=coredns-7448499f4d-f7kjn_kube-system(12a89af8-4450-4ae2-92d1-97fa2d32d1c0)\"" pod="kube-system/coredns-7448499f4d-f7kjn" podUID=12a89af8-4450-4ae2-92d1-97fa2d32d1c0
~

@lauchokyip
Copy link
Author

Logs from worker node

Jun 02 13:51:40 knode0 systemd[1]: Starting Lightweight Kubernetes...
Jun 02 13:51:41 knode0 systemd[1]: Started Lightweight Kubernetes.
Jun 02 13:51:45 knode0 k3s[464]: time="2021-06-02T13:51:45.538548160-04:00" level=info msg="Starting k3s agent v1.21.1+k3s1 (75dba57f)"
Jun 02 13:51:45 knode0 k3s[464]: time="2021-06-02T13:51:45.542379457-04:00" level=info msg="Running load balancer 127.0.0.1:6444 -> [[2601:152:4001:aa90::1e]:6443 192.168.0.21:6443]"
Jun 02 13:51:45 knode0 k3s[464]: time="2021-06-02T13:51:45.562536660-04:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Jun 02 13:51:47 knode0 k3s[464]: time="2021-06-02T13:51:47.568689252-04:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Jun 02 13:52:00 knode0 k3s[464]: time="2021-06-02T13:52:00.031077263-04:00" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Jun 02 13:52:05 knode0 k3s[464]: time="2021-06-02T13:52:05.200307908-04:00" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Jun 02 13:54:11 knode0 k3s[464]: time="2021-06-02T13:54:11.606726381-04:00" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.658250931-04:00" level=info msg="Module overlay was already loaded"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.706831708-04:00" level=info msg="Module br_netfilter was already loaded"
Jun 02 13:54:20 knode0 k3s[464]: W0602 13:54:20.774633     464 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.778751468-04:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.779087320-04:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.779633227-04:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.780054597-04:00" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.805748005-04:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Jun 02 13:54:20 knode0 k3s[464]: time="2021-06-02T13:54:20.807987449-04:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Jun 02 13:54:21 knode0 k3s[464]: time="2021-06-02T13:54:21.820761874-04:00" level=info msg="Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet"
Jun 02 13:54:22 knode0 k3s[464]: time="2021-06-02T13:54:22.828108781-04:00" level=info msg="Containerd is now running"
Jun 02 13:54:23 knode0 k3s[464]: time="2021-06-02T13:54:23.290214688-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:23 knode0 k3s[464]: time="2021-06-02T13:54:23.308060225-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:23 knode0 k3s[464]: time="2021-06-02T13:54:23.308330281-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:28 knode0 k3s[464]: time="2021-06-02T13:54:28.309426500-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:28 knode0 k3s[464]: time="2021-06-02T13:54:28.319349130-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:28 knode0 k3s[464]: time="2021-06-02T13:54:28.319431685-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:33 knode0 k3s[464]: time="2021-06-02T13:54:33.319903145-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:33 knode0 k3s[464]: time="2021-06-02T13:54:33.354031108-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:33 knode0 k3s[464]: time="2021-06-02T13:54:33.354205682-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:38 knode0 k3s[464]: time="2021-06-02T13:54:38.355053198-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:38 knode0 k3s[464]: time="2021-06-02T13:54:38.378996791-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:38 knode0 k3s[464]: time="2021-06-02T13:54:38.379169272-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:43 knode0 k3s[464]: time="2021-06-02T13:54:43.380420363-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:43 knode0 k3s[464]: time="2021-06-02T13:54:43.404385369-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:43 knode0 k3s[464]: time="2021-06-02T13:54:43.404663227-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:48 knode0 k3s[464]: time="2021-06-02T13:54:48.404975100-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:48 knode0 k3s[464]: time="2021-06-02T13:54:48.422915765-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:48 knode0 k3s[464]: time="2021-06-02T13:54:48.423085601-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:53 knode0 k3s[464]: time="2021-06-02T13:54:53.423680364-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:53 knode0 k3s[464]: time="2021-06-02T13:54:53.441450279-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:53 knode0 k3s[464]: time="2021-06-02T13:54:53.441991471-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:58 knode0 k3s[464]: time="2021-06-02T13:54:58.442807195-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:54:58 knode0 k3s[464]: time="2021-06-02T13:54:58.452959672-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:54:58 knode0 k3s[464]: time="2021-06-02T13:54:58.453047654-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:03 knode0 k3s[464]: time="2021-06-02T13:55:03.454165892-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:03 knode0 k3s[464]: time="2021-06-02T13:55:03.477832317-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:03 knode0 k3s[464]: time="2021-06-02T13:55:03.478092689-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:08 knode0 k3s[464]: time="2021-06-02T13:55:08.479109358-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:08 knode0 k3s[464]: time="2021-06-02T13:55:08.496237695-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:08 knode0 k3s[464]: time="2021-06-02T13:55:08.497215440-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:13 knode0 k3s[464]: time="2021-06-02T13:55:13.498163453-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:13 knode0 k3s[464]: time="2021-06-02T13:55:13.521621382-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:13 knode0 k3s[464]: time="2021-06-02T13:55:13.521949624-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:18 knode0 k3s[464]: time="2021-06-02T13:55:18.523173453-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:18 knode0 k3s[464]: time="2021-06-02T13:55:18.539444973-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:18 knode0 k3s[464]: time="2021-06-02T13:55:18.539619288-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:23 knode0 k3s[464]: time="2021-06-02T13:55:23.540595597-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:23 knode0 k3s[464]: time="2021-06-02T13:55:23.562748857-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:23 knode0 k3s[464]: time="2021-06-02T13:55:23.562922154-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:28 knode0 k3s[464]: time="2021-06-02T13:55:28.563913258-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:28 knode0 k3s[464]: time="2021-06-02T13:55:28.577456460-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:28 knode0 k3s[464]: time="2021-06-02T13:55:28.577551145-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:33 knode0 k3s[464]: time="2021-06-02T13:55:33.578229477-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:33 knode0 k3s[464]: time="2021-06-02T13:55:33.594800455-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:33 knode0 k3s[464]: time="2021-06-02T13:55:33.594974140-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:38 knode0 k3s[464]: time="2021-06-02T13:55:38.596244959-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:38 knode0 k3s[464]: time="2021-06-02T13:55:38.612025840-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:38 knode0 k3s[464]: time="2021-06-02T13:55:38.612199692-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:43 knode0 k3s[464]: time="2021-06-02T13:55:43.612499666-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:43 knode0 k3s[464]: time="2021-06-02T13:55:43.633839861-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:43 knode0 k3s[464]: time="2021-06-02T13:55:43.634020695-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:48 knode0 k3s[464]: time="2021-06-02T13:55:48.634478098-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:48 knode0 k3s[464]: time="2021-06-02T13:55:48.650375835-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:48 knode0 k3s[464]: time="2021-06-02T13:55:48.650548323-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:53 knode0 k3s[464]: time="2021-06-02T13:55:53.651560544-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:53 knode0 k3s[464]: time="2021-06-02T13:55:53.673772419-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:53 knode0 k3s[464]: time="2021-06-02T13:55:53.673964815-04:00" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"
Jun 02 13:55:58 knode0 k3s[464]: time="2021-06-02T13:55:58.674881671-04:00" level=info msg="Connecting to proxy" url="wss://[2601:152:4001:aa90::1e]:6443/v1-k3s/connect"
Jun 02 13:55:58 knode0 k3s[464]: time="2021-06-02T13:55:58.690360245-04:00" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.0.21, not 2601:152:4001:aa90::1e"

@brandond
Copy link
Member

brandond commented Jun 2, 2021

It looks like pods on the worker are unable to communicate with services hosted on the server. I see a lot of IPv6 addresses in both the pod logs and the agent logs, but the flannel CNI does not support IPv6 at the moment. Can you try disabling IPv6 on both nodes, and then rebooting?

@lauchokyip
Copy link
Author

Thanks @brandond , disabled IPv6 following this guide and seems to be working now 👍🏼 😃

@criscola
Copy link

criscola commented Jun 29, 2021 •
edited
Loading

This is not a solution tho, it's a workaround disabling IPv6, which could be needed by some users... it looks like flannel's contributors are working on it right now: flannel-io/flannel#1448

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brandond @criscola @lauchokyip