Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provide workarounds for the kubelet self-signed serving certificate #1602

Closed
neolit123 opened this issue Jun 11, 2019 · 15 comments · Fixed by kubernetes/website#27071
Closed
Assignees
Labels
area/security kind/documentation Categorizes issue or PR as related to documentation. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@neolit123
Copy link
Member

neolit123 commented Jun 11, 2019

TL;DR kubeadm manages a kubelet that self-signs it's serving certificate. this creates a blocker for metric server users that wish to scrape the kubelet as a server.

related issue:
#1223

related enhancement:
kubernetes/enhancements#267

official docs:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation

Note: The CSR approving controllers implemented in core Kubernetes do not approve node serving certificates for security reasons. To use RotateKubeletServerCertificate operators need to run a custom approving controller, or manually approve the serving certificate request

this ticket is to track documenting a workaround in our TS guide:
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/

or alternatively as a MD file / guide in this repository under /docs.

@neolit123 neolit123 added area/security priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. kind/documentation Categorizes issue or PR as related to documentation. labels Jun 11, 2019
@neolit123 neolit123 added this to the v1.16 milestone Jul 2, 2019
@neolit123
Copy link
Member Author

someone shared this operator:
#1635 (comment)

@fabriziopandini
Copy link
Member

@neolit123 is #1753 going to fix this as well?

@neolit123
Copy link
Member Author

neolit123 commented Sep 16, 2019

@fabriziopandini no, this is about the kubelet self-signed serving cert.
some users want it signed with the CA, instead.

@neolit123 neolit123 modified the milestones: v1.16, v1.17 Sep 23, 2019
@NeilW
Copy link

NeilW commented Nov 4, 2019

Helm workaround is to turn off TLS checking for the metrics server

$ helm install --set 'args={--kubelet-insecure-tls}' --namespace kube-system metrics stable/metrics-server

@neolit123 neolit123 removed their assignment Nov 9, 2019
@neolit123 neolit123 added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Nov 9, 2019
@neolit123 neolit123 modified the milestones: v1.17, v1.18 Nov 9, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 7, 2020
@neolit123
Copy link
Member Author

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 7, 2020
@zedtux
Copy link

zedtux commented Feb 17, 2020

Looking for this so that deploying metrics-server would work securely and out-of-the-box!

@blepoutr
Copy link

blepoutr commented Sep 8, 2020

Hello

Looking for this so that deploying metrics-server would work securely and out-of-the-box!
Indeed, and also to allow proper APIserver to kubelet configuration ( which is now not validating Kubelet serving cert with any CA per doc : https://v1-16.docs.kubernetes.io/docs/concepts/architecture/master-node-communication/#apiserver-to-kubelet )

@NeilW
Copy link

NeilW commented Sep 15, 2020

I've written up the process and how to get metrics server to work securely with a kubeadm deployed Kubernetes system here:

https://www.brightbox.com/blog/2020/09/15/secure-kubernetes-metrics/

Hope you find it useful.

@neolit123
Copy link
Member Author

I've written up the process and how to get metrics server to work securely with a kubeadm deployed Kubernetes system here:

https://www.brightbox.com/blog/2020/09/15/secure-kubernetes-metrics/

Hope you find it useful.

Thanks for the writeup, Neil.
Related to 'serverTLSBootstrap: true' it is important to note that this certificate will expire after 1 year. If serving cerificate rotation is enabled, the user needs to approve the CSR when that happens. Thus, users created the controllers mentioned above.

I wish we had a guide for using rbac proxy too.

@blepoutr
Copy link

Hello @NeilW

Thanks for the very good article .
One variant seems to sign the Serving/Server csr of the Metrics server with Kubernetes.
See https://github.com/jenting/secure-metrics-server/blob/master/gen-metrics-server-cert-key.sh.

@NeilW
Copy link

NeilW commented Sep 24, 2020

I've added another post about auto-signing the kubelet certificates using "rubber stamp" and why we believe that is an acceptable risk on kubeadm installed clusters.

https://www.brightbox.com/blog/2020/09/24/auto-signing-kubernetes-server-certificates/

@randomvariable
Copy link
Member

The situation has changed somewhat since this was opened.

The current state is that cloud providers are or are going to implement CSR signers that verify the identity of the node via an out of band mechanism, and that would provide a stronger guarantee than the rubber stamp controller alone.

We should probably direct users to look at the relevant cloud provider docs to see what their options are, and suggest they can use a rubber stamp or custom controller given the subject access review etc...

@neolit123 neolit123 modified the milestones: Next, v1.21 Jan 21, 2021
@neolit123
Copy link
Member Author

i can summarize the state in the docs this cycle.

@neolit123 neolit123 added lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. and removed help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. labels Mar 9, 2021
@neolit123
Copy link
Member Author

opened PR for the kubeadm docs:
kubernetes/website#27071

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security kind/documentation Categorizes issue or PR as related to documentation. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants