Adding Kubelet Serving Certificate using serverTLSBootstrap: true

[munai-das]

What keywords did you search in kubeadm issues before filing this one?

Versions

kubeadm version (use kubeadm version): v1beta1

Environment:

• Kubernetes version (use kubectl version): kubernetes 1.14.1
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
• Kernel (e.g. uname -a): 3.10.0-957.21.3.el7.x86_64
• Others:
  Container-Runtime: docker://18.6.2

What happened?

Added the below to make Kubelet Serving Certificate work.

As a part of KubeletConfiguration in master0(3 master setup)

rotateCertificates: true
featureGates:
  RotateKubeletClientCertificate: true
  RotateKubeletServerCertificate: true
serverTLSBootstrap: true

& kubelet-certificate-authority: "/etc/kubernetes/pki/ca.crt" to kube-apiserver

Additionally, I added below to nodeRegistration of worker JoinConfiguration

rotate-certificates: "true"
    rotate-server-certificates: "true"
    feature-gates: "RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true"

Moreover, I also made sure that I added -node-ip to all my kubelets.
(Refering https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#non-public-ip-used-for-containers)
I forked and modified CoreOS’ kapprovers to auto-approve CSR requests by kubelets.

What you expected to happen?

Kubelet Serving Certificates to be issued by CSR API.

kubectl logs stopped working with the error
Error from server: Get https://XXXX:10250/containerLogs/: dial tcp XXXX:10250: connect: no route to host

How to reproduce it (as minimally and precisely as possible)?

Add the flags as described in What Happened above.

Anything else we need to know?

No.

[neolit123]

/triage support
/area security

hi, this doesn't look like a kubeadm problem, per se.
kubeadm already provides you with all the settings to use the RotateKubeletServerCertificate feature.

have you looked at the kubelet logs for any errors?

[neolit123]

i think the CSR is not being issued because the certificate is still being self-signed:
https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration

// tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert,
    // if any, concatenated after server cert). If tlsCertFile and
    // tlsPrivateKeyFile are not provided, a self-signed certificate
    // and key are generated for the public address and saved to the directory
    // passed to the Kubelet's --cert-dir flag.
    // Dynamic Kubelet Config (beta): If dynamically updating this field, consider that
    // it may disrupt components that interact with the Kubelet server.

try passing those settings too.
i haven't played with serverTLSBootstrap.

[munai-das]

Thanks for the comment. I will check this.

[munai-das]

The issue was --node-ip not getting added to two of the masters. As it is not possible to add via KubeletConfiguration, I added it using sed. Closing the issue.

[funkypenguin]

@munai-das I'm trying to do the same thing - can you share your changes to kapprovers?

[munai-das]

I have internally refactored the kapprovers piece. I will ask the concerned person in the office and open source it.

[funkypenguin]

Thanks, but I sorted it with https://github.com/kontena/kubelet-rubber-stamp ;) Cheers! D On 9 July 2019 at 19:08:38, Munai Das Udasin ([email protected]) wrote: I have internally refactored the kapprovers piece. I will ask the concerned person in the office and open source it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1635?email_source=notifications&email_token=AALUHTWUMJ5KT6FX4UCED7LP6Q2PLA5CNFSM4H3BKW4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPKGPQ#issuecomment-509518654>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALUHTWEY6PFNXW3HNVIBILP6Q2PLANCNFSM4H3BKW4A> .