Kubeadm 1.15.2 upgrade fails #2352

ttrumm · 2020-11-16T10:25:21Z

What keywords did you search in kubeadm issues before filing this one?

nodes "kube-apiserver-kubelet-client" not found
[upgrade/config] FATAL: failed to get node registration: failed to get corresponding node: nodes "kube-apiserver-kubelet-client" not found

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:20:51Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration:
Azure
OS (e.g. from /etc/os-release):
Centos 7.2
Kernel (e.g. uname -a):
Linux kube-master-we 3.10.0-1062.12.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Others:

What happened?

Master node name kube-master-we

[root@kube-master-we tmp]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: failed to get node registration: failed to get corresponding node: nodes "kube-apiserver-kubelet-client" not found

What you expected to happen?

cd /tmp/

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

kubeadm upgrade plan --config /tmp/kubeadm.yaml
[upgrade/config] Making sure the configuration is correct:
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.2
[upgrade/versions] kubeadm version: v1.15.2

How to reproduce it (as minimally and precisely as possible)?

Created local master node with Vagrant

Create cluster and get current date

[root@node1 ~]# date
Mon Nov 16 11:24:21 UTC 2020

kubeadm init --token-ttl=0 --apiserver-advertise-address=172.17.8.101 --pod-network-cidr=172.17.0.0/16 --apiserver-cert-extra-sans=node1,172.17.8.101 --ignore-preflight-errors=NumCPU

mkdir -p .kube && cp -i /etc/kubernetes/admin.conf /root/.kube/config

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" && touch /root/.weave_done

Lets go In future and check certificate expiration date

[root@node1 ~]# date -s "1 OCT 2021 18:00:00"
Fri Oct  1 18:00:00 UTC 2021

[root@node1 ~]# kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 16, 2021 11:12 UTC   45d             no
apiserver                  Nov 16, 2021 11:12 UTC   45d             no
apiserver-etcd-client      Nov 16, 2021 11:12 UTC   45d             no
apiserver-kubelet-client   Nov 16, 2021 11:12 UTC   45d             no
controller-manager.conf    Nov 16, 2021 11:13 UTC   45d             no
etcd-healthcheck-client    Nov 16, 2021 11:12 UTC   45d             no
etcd-peer                  Nov 16, 2021 11:12 UTC   45d             no
etcd-server                Nov 16, 2021 11:12 UTC   45d             no
front-proxy-client         Nov 16, 2021 11:12 UTC   45d             no
scheduler.conf             Nov 16, 2021 11:13 UTC   45d             no

Lets renew certs in future

[root@node1 ~]# kubeadm alpha certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healtcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Lets go more into future

[root@node1 ~]# date -s "20 NOV 2021 18:00:00"
Sat Nov 20 18:00:00 UTC 2021

[root@node1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: failed to get config map: Get https://172.17.8.101:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: x509: certificate has expired or is not yet valid

Fix 1.15 kubelet.conf issue #2185

mkdir /tmp/certs
cd /tmp/certs/
cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet-backup
cp /etc/kubernetes/pki/apiserver-kubelet-client.* .
cat apiserver-kubelet-client.crt | base64 > cert-base64
cat apiserver-kubelet-client.key | base64 > key-base64

copy base64 values into /etc/kubernetes/kubelet.conf 
cp -i /etc/kubernetes/admin.conf /root/.kube/config
reboot

Reproduced issue

[root@node1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: failed to get node registration: failed to get corresponding node: nodes "kube-apiserver-kubelet-client" not found

Workaround

cd /tmp/

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

kubeadm upgrade plan --config /tmp/kubeadm.yaml
[upgrade/config] Making sure the configuration is correct:
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.2
[upgrade/versions] kubeadm version: v1.15.2

The text was updated successfully, but these errors were encountered:

ttrumm · 2020-11-16T12:35:52Z

New reproduce with kubeadm 1.16
Kubeadm 1.16.15-0

Create cluster

kubeadm init --token-ttl=0 --apiserver-advertise-address=172.17.8.101 --pod-network-cidr=172.17.0.0/16 --apiserver-cert-extra-sans=node1,172.17.8.101 --ignore-preflight-errors=NumCPU

mkdir -p .kube && cp -i /etc/kubernetes/admin.conf /root/.kube/config

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" && touch /root/.weave_done

Get current date

[root@node1 ~]# date
Mon Nov 16 12:28:13 UTC 2020

Lets go 11 months inyo future

[root@node1 ~]# date -s "1 OCT 2021 18:00:00"
Fri Oct  1 18:00:00 UTC 2021

[root@node1 ~]# kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 16, 2021 12:26 UTC   45d             no
apiserver                  Nov 16, 2021 12:26 UTC   45d             no
apiserver-etcd-client      Nov 16, 2021 12:26 UTC   45d             no
apiserver-kubelet-client   Nov 16, 2021 12:26 UTC   45d             no
controller-manager.conf    Nov 16, 2021 12:26 UTC   45d             no
etcd-healthcheck-client    Nov 16, 2021 12:26 UTC   45d             no
etcd-peer                  Nov 16, 2021 12:26 UTC   45d             no
etcd-server                Nov 16, 2021 12:26 UTC   45d             no
front-proxy-client         Nov 16, 2021 12:26 UTC   45d             no
scheduler.conf             Nov 16, 2021 12:26 UTC   45d             no

Renew Certificates in future

[root@node1 ~]# kubeadm alpha certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Lets go more into future

[root@node1 ~]# date -s "20 NOV 2021 18:00:00"
Sat Nov 20 18:00:00 UTC 2021

[root@node1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: failed to get config map: Get https://172.17.8.101:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: x509: certificate has expired or is not yet valid
To see the stack trace of this error execute with --v=5 or higher

Fix kubelet.conf with new certificates

[root@node1 certs]# kubeadm upgrade plan
[root@node1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: failed to get node registration: failed to get corresponding node: nodes "kube-apiserver-kubelet-client" not found
To see the stack trace of this error execute with --v=5 or higher

ttrumm · 2020-11-16T12:53:56Z

With 1.17 kubeadm it is ok. From 1.17 kubeadm alpha cert renew upgrades kubelet.conf itself. #1753

[root@node1 ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Nov 16, 2021 12:46 UTC   45d                                     no
apiserver                  Nov 16, 2021 12:46 UTC   45d             ca                      no
apiserver-etcd-client      Nov 16, 2021 12:46 UTC   45d             etcd-ca                 no
apiserver-kubelet-client   Nov 16, 2021 12:46 UTC   45d             ca                      no
controller-manager.conf    Nov 16, 2021 12:46 UTC   45d                                     no
etcd-healthcheck-client    Nov 16, 2021 12:46 UTC   45d             etcd-ca                 no
etcd-peer                  Nov 16, 2021 12:46 UTC   45d             etcd-ca                 no
etcd-server                Nov 16, 2021 12:46 UTC   45d             etcd-ca                 no
front-proxy-client         Nov 16, 2021 12:46 UTC   45d             front-proxy-ca          no
scheduler.conf             Nov 16, 2021 12:46 UTC   45d                                     no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Nov 14, 2030 12:46 UTC   9y              no
etcd-ca                 Nov 14, 2030 12:46 UTC   9y              no
front-proxy-ca          Nov 14, 2030 12:46 UTC   9y              no

Lets renew certs in future

[root@node1 ~]# kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Lets go more into future

date -s "20 NOV 2021 18:00:00"

[root@node1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.17.14
[upgrade/versions] kubeadm version: v1.17.14
W1120 18:00:26.377242    2353 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: x509: certificate has expired or is not yet valid
W1120 18:00:26.377353    2353 version.go:102] falling back to the local client version: v1.17.14
[upgrade/versions] Latest stable version: v1.17.14
W1120 18:00:26.451517    2353 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.17.txt": Get https://dl.k8s.io/release/stable-1.17.txt: x509: certificate has expired or is not yet valid
W1120 18:00:26.451583    2353 version.go:102] falling back to the local client version: v1.17.14
[upgrade/versions] Latest version in the v1.17 series: v1.17.14

Awesome, you're up-to-date! Enjoy!
[root@node1 ~]# date
Sat Nov 20 18:02:02 UTC 2021

neolit123 · 2020-11-16T15:43:46Z

hello, first of all older versions than 1.17 are already out of support, and 1.17 will be out of support soon with the release of 1.20.
we cannot backport fixes for versions older than 1.17 because they are no longer released!

a couple of points:

With 1.17 kubeadm it is ok. From 1.17 kubeadm alpha cert renew upgrades kubelet.conf itself. #1753

kubeadm renew does not touch kubelet.conf, it is the responsibility of "kubeadm init" to prepare kubelet.conf to point to cert files instead of embedding certs.

Lets go more into future

note that when you are doing that artificially, you might trip the kubelet cert renewal mechanism and the files in /var/lib/kubelet/pki might not get renewed properly.

if 1.17 works for you just try to upgrade to this version, but then also upgrade to 1.18.
if these newer versions fail, only then we can take action.

let me know if you have more questions.

/kind support
/close

k8s-ci-robot · 2020-11-16T15:44:02Z

@neolit123: Closing this issue.

In response to this:

hello, first of all older versions than 1.17 are already out of support, and 1.17 will be out of support soon with the release of 1.20.
we cannot backport fixes for versions older than 1.17 because they are no longer released!

a couple of points:

With 1.17 kubeadm it is ok. From 1.17 kubeadm alpha cert renew upgrades kubelet.conf itself. #1753

kubeadm renew does not touch kubelet.conf, it is the responsibility of "kubeadm init" to prepare kubelet.conf to point to cert files instead of embedding certs.

Lets go more into future

note that when you are doing that artificially, you might trip the kubelet cert renewal mechanism and the files in /var/lib/kubelet/pki might not get renewed properly.

if 1.17 works for you just try to upgrade to this version, but then also upgrade to 1.18.
if these newer versions fail, only then we can take action.

let me know if you have more questions.

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zhegemingzimeibanquan · 2024-01-19T07:09:10Z

at the end，how do you fix this problem,i meet the same promblem now,i don`t know how the node appear

when i run the kubectl get node ,there is not the "kubernetes-admin" node

k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Nov 16, 2020

k8s-ci-robot closed this as completed Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubeadm 1.15.2 upgrade fails #2352

Kubeadm 1.15.2 upgrade fails #2352

ttrumm commented Nov 16, 2020 •

edited

Loading

ttrumm commented Nov 16, 2020 •

edited

Loading

ttrumm commented Nov 16, 2020 •

edited

Loading

neolit123 commented Nov 16, 2020

k8s-ci-robot commented Nov 16, 2020

zhegemingzimeibanquan commented Jan 19, 2024

Kubeadm 1.15.2 upgrade fails #2352

Kubeadm 1.15.2 upgrade fails #2352

Comments

ttrumm commented Nov 16, 2020 • edited Loading

What keywords did you search in kubeadm issues before filing this one?

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Workaround

ttrumm commented Nov 16, 2020 • edited Loading

ttrumm commented Nov 16, 2020 • edited Loading

neolit123 commented Nov 16, 2020

k8s-ci-robot commented Nov 16, 2020

zhegemingzimeibanquan commented Jan 19, 2024

ttrumm commented Nov 16, 2020 •

edited

Loading

ttrumm commented Nov 16, 2020 •

edited

Loading

ttrumm commented Nov 16, 2020 •

edited

Loading