-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
metrics-server error because it doesn't contain any IP SANs #196
Comments
I'm use the metrics-server version is v0.2.1 |
v0.3.1 the same problem command: |
please make sure that your certificates are signed for the right names, and that your nodes are surfacing those names. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Did you do something like this :
Remove |
Actually in the new version, is just add it to args
Thank you, solved my problem! |
same problem. "--kubelet-insecure-tls" solved it. |
--kubelet-insecure-tls is just for testing purpose only. it just skip the issue, not resolve it totally. |
Why is this issue still happening, why is the workaround still required? I Just setup a new cluster (k8s 1.20.6) from scratch (Ubuntu 20.04 LTS) and when I deployed the metrics server, bang, bug. |
The default configuration of metrics assumes that you have cluster setup with reasonable (not in any way great) security, basic certificate setup. This is pretty good assumption for most production clusters. On the other hand most local development tools like (minikube, kind, k3d etc) skip proper certificate setup, requiring them to pass Having default secure setup, and simple way to disable it for local development seems like a good default, but let me know if you think otherwise. |
Hmm, I setup the cluster via kubeadm, so security should be setup properly. I would understand if minkube and kind would do shortcuts on security, they are not meant for productive use, but kubeadm should not. |
I agree with @raspitakesovertheworld. I did the same, with kubeadm, and only ran into problems with the metrics server ;-( |
Same thing here, I used kubeadm for bootstraping K8s v21 cluster and ran to the same issue |
Who said that the default kubeadm configuration is secure? |
No one .. Logs from the metrics Pod:
|
To avoid passing |
Actually I was also looking at this document, but they wrote:
But in a kubeadm deployed cluster, there is no
Which you can not edit as they described |
Please file an issue to KubeAdm. |
thanks! |
This is no solution but a security issue waiting to happen. Anyone interested in a actual solution still or do we simply disable TLS verification everywhere now gals? The problem seems to be that metrics is using the IPs to communicate with the nodes. It's totaly legit that the node certificates don't contain IPs in the SAN since this is considered bad security practise (see RFC 6125 for more details). |
/reopen |
@omniproc: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Looking more into this the problem seems to be that by default this order is used for the metrics server:
However the docs clearly say the default order should be like this:
Quote from the docs
explicitly setting this as arg for the metrics-server seems to actually fix the issue. |
Part of the docs you are referring to is about command line flags. It's true that command line flags defaults doesn't match does we use in manifests. We changed the config as it supports more clusters out of the box. This is because most cluster distributions configure Nodes with Hostname as address, but it is not resolvable domain by cluster DNS. We also found that using DNS can be problematic as it can be unreliable, we preferred to avoid dependency on DNS. Using IP addresses meant that we cannot validate Kubelet certificates, but we found out that most K8s distros don't provide Kubelets with certificates signed by k8s CA, meaning users would still need to disable validating certs. So in the end default configuration is not most secure one as it requires a lot of preparation by cluster administrator to reconfigure cluster level configuration to work. Work which most users don't want to do, or just cant as they cannot change cluster config. Overall topic of MS security is very broad topic as there are a lot of other places that wound need improvements (#576 #545) however there was little interest in addressing those issues by Security folks as attack vector via MS is not very critical. Basically attacker could inject an invalid metric value into HPA that could cause a unneeded scale up. However, if you are interested in this topic and would like to contribute security improvements I would be happy help with that. |
OK I get the reasoning behind the decission to use that as default values. The thing that bothers me is that the implications aren't communicated well by the docs.
I agree that ultimatly it's the responsibility of the administrator who runs metrics to take care about those things but it can't hurt to warn / remind people even more so when every google search so far leads the unaware admin to think disabling TLS is the solution. |
For me what solved the issue was: https://particule.io/en/blog/kubeadm-metrics-server/ Long story short:
|
You said something but said nothing😅 |
i am curious to know what The Right Way to fix this is its definitely possible to work around the issue with it seems like after reading #196 (comment) i think i understand what the situation (i think) tl;dr, there is a Right Way, but its kind of complicated, and leaving it insecure isn't that big of a deal so just use OR DONT!! https://particule.io/en/blog/kubeadm-metrics-server/ does work! this is basically a copy of a section in kubernetes documentation: Certificate Management with
|
Thanks! It WORKS! |
I'm having the same error with Kubernetes metrics-server version 0.6.1 running on Linode Kubernetes Engine (LKE) k8s version 1.23. Installation command:
In the Metrics Server Pod logs, I just see this:
|
I'd only like to mention that the article contains a piece of advice to edit Thus, one can make changes to Cheers 🍻 |
Obviously not for production but if you just have a local dev cluster you created with kubectl -n kube-system patch deployment metrics-server --type=json \
-p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]]' |
@awalker125 Works like a charm! Changes are not applied immediately, you still need to restart the metrics server pod to read the patched config map :
|
I solve it in KinD, and write a detailed post https://www.zeng.dev/post/2023-kubeadm-enable-kubelet-serving-certs/ |
kubernetes-sigs/metrics-server#196 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#cannot-use-the-metrics-server-securely-in-a-kubeadm-cluster https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely siderolabs/talos#3502 siderolabs/talos#3166
Brilliant @phosae, thanks for the detailed solution for kind. |
My solution for a running cluster with metric server already installed (I installed it with helm). On the control plane: k -n kube-system edit configmap kubelet-config Append On each node: sudo nano /var/lib/kubelet/config.yaml Append
Verify:
Done. |
This worked for my local kind setup. Thank you @phosae |
The case is, when you add - --kubelet-insecure-tls, it does not matter what the secure port is, so this configuration works for me: |
Definite answer for people having setup their cluster with
|
I am having this issue on a "Kubernetes the hard way" cluster that i created from scratch without kubeadm, kind, helm or the like. |
when I finish deploy the metrics-server ,Go to the log and find the following error
E1229 07:09:05.013998 1 summary.go:97] error while getting metrics summary from Kubelet kube-node3(172.16.52.132:10250): Get https://172.16.52.132:10250/stats/summary/: x509: cannot validate certificate for 172.16.52.132 because it doesn't contain any IP SANs
The text was updated successfully, but these errors were encountered: