Metrics server issue with hostname resolution of kubelet and apiserver unable to communicate with metric-server clusterIP #131
Comments
|
@vikranttkamble you can try --kubelet-preferred-address-types InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP |
|
I'm having the same issue. +1 |
|
I think the main problem is that the hostname resolution is being performed through the internal DNS server (which is set by default to the pod where the metrics-server runs in). That server contains the pods/services entries, but not the cluster-nodes ones. AFAIK the cluster nodes are not in that scope so they can't be resolved via that DNS. The InternalIP should be queried from the api instead |
|
The solution proposed by @MIBc works. Change the metrics-server-deployment.yaml file and add: The metrics-server is now able to talk to the node (It was failing before because it could not resolve the hostname of the node). There's something strange happening though, I can see the metrics now from HPA but it shows an error in the logs: |
|
I and Vikrant is working same servers, We are now able to edit Metric Server from below command.
|
|
In Metrics Servers we are found below logs.
|
|
It is OK when set the kubelet flag "--authorization-mode=AlwaysAllow", and metrics-server flag "--kubelet-insecure-tls". |
|
I think it should authorize for metrics-server to access kubelet if authorization-mode=webhook. |
|
We also got, SSL issue and get sock connection refused issue, and resolved with below conf parameters in metrics-server-deployment.yaml We are currently facing proxy issue and working on it. |
|
@MIBc Also InternalIP we have to put IP or just the keep it InternalIP? |
|
@amolredhat The '--source' flag is unavailable right now (v0.3.0-alpha.1) I (finally) got it to work by setting the following args: command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIPIt works like a charm! |
|
@juan-vg awesome, this also works for me too (metrics-server-amd64:v.0.3.0 on k8s 1.10.3). Btw, so as not to duplicate the entrypoint set in the Dockerfile, consider using args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP |
|
If you're going to use InternalIP, you should probably set up your node's serving certs to list the IP as an alternative name. You generally don't want to pass |
|
This seems to be a blocking issue for 0.3.0 when running a kops deployment on AWS using a private network topology. Naturally kubedns can't resolve that hostname. I tried setting Not really sure what to do with that. I don't want to start monkeying with my node's certs without knowing exactly what I'm fixing. For now I've had to revert to metrics-server 0.2.1. |
|
You're the second person to mention issues with kops (#133), so I'm starting to think that kops sets up its certs differently than expected. Basically, the issue is that whatever the kops kubelet serving certs are, they aren't signed by the default kubernetes CA. Can we maybe get a kops maintainer in here to comment? |
|
@wilsonjackson @DirectXMan12 And it Worked !! Also, we changed some parameters in kubernetes/manifests/kube-apiserver.yaml |
|
I temporary fix this by edit metric-service deployemnt add this cofnig unser Deployemnt.spec.template.spec |
|
Obviously the issue in In my case coreDNS is using for cluster DNS resolutions. Then we could look at default config for coreDNS This option So, yes, I've looked in my DNS logs and found that my DNS were received that requests. Eventually, I've just added my node's hostnames records to my external DNS service and that's it. |
|
Awesome. I'm going to close this issue, but feel free to ping me if you think it's not solved yet. |
Hi, I'm using kube-dns instead of coredns, and I have my node's /etc/hosts set properly, and it still fails: |
The host uses /etc/hosts to parse. How to handle it better? |
问题解决了吗？ |
Hi, I can assume your nslookup request will fail. |
|
|
@originsmike Another problem after modify tls and internalIP |
|
I am facing bit different issue here. Don't know if it is metrics server problem or api server. But thought of posting here. Please see below command output and logs and let me know whts wrong. I believe ApiServer is not able to contact metrics-server over internal IPs. # kubectl -n kube-system describe apiservice v1beta1.metrics.k8s.io # kubectl -n kube-system get svc -o wide Logs from APIServer E1020 17:56:51.069077 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.99.121.153:443: dial tcp 10.99.121.153:443: connect: no route to host |
Would you please provide some details on how to do it? |
|
@DirectXMan12 , |
|
@silverbackdan Hopefully I can help. I assume you already know about CSRs. You can start with https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request-object-to-send-to-the-kubernetes-api. Once you approve and download the cert, put the private key from your CSR in /var/lib/kubelet/pki/kubelet.key and the cert in /var/lib/kubelet/pki/kubelet.crt. Then restart kubelet service. you are good to go. |
|
Thanks very much for this info. Just to check, I'd be replacing the key and cert already at those paths because they do exist. I imagine they are just self-signed which is what is being used at the moment and therefore is seen as insecure? |
|
Yes, that is correct. They are self-signed certs and you will be replacing them with certs signed by your Kubernetes cluster CA. I wish the |
|
Also, I wonder if this would avoid the issue - https://kubernetes.io/docs/tasks/tls/certificate-rotation/. |
|
Perhaps, specifically the I had to run this command to change the cert SANs for the API and to listen on different IPs kubeadm init phase certs all \
--apiserver-advertise-address=0.0.0.0 \
--apiserver-cert-extra-sans=10.244.0.1,11.0.0.10,example.comIt'd have been nice if that command could also sign the cert with the cluster's CA, anyway, I see where things are going wrong now and can use your very helpful notes to take me further. |
|
Nice! I was reading that same page too. There might be more learnings on the page you linked for me too. Queued for when I have some learning time. :-) |
|
I have the same issue resolved by adding : Whereas the metrics server now return only used ressources for The master node (always unknown for workers node). NB. Deployment done with vagrant (hence kubebadm). Please have you any hint to solve it |
|
I'm still getting the error, I'm using Kubespray v2.11.0 to deploy a Kubernetes v1.15.3 with the following instruction: |
I just got the same situation on my metrics-server,unable to fetch pod metrics for pod default/xxxxxxxx: no metrics known for pod "default/xxxxxxxx" @damascenorakuten |
|
@carlylelu appirence of this log does not mean that there is a problem, nor points to possible cause. |
|
Got the same issue (host resolution) when installing server-metrics in a DO cluster. I think I did not have the issue a couple of days ago :( |
|
One small detail that may get overlooked, in case a proxy is used, that could be caused by forgetting to add the 10.0.0.0/8 to the no_proxy list. |
|
In recent versions of metrics-server, where there is no "command" or "metrics-server-deployment.yaml", the following helped me
|
|
@veton 's answer is the most on-point and up to date, thank you! 🙏 |
Thanks, that solved my problem too |
|
Adding |
I tried this suggestion and it worked for me. My problem: Here is my working yaml file: You can compare to the one in GitHub Note: the IP addresses I used for the hostAliases are the ones that were assigned to the nodes for the pod network by Flannel, so they are reachable from all of the containers. |
|
[root@k8s-master1:~]# kubectl get svc -n kube-system -o wide [root@k8s-master1:~]# kubectl describe apiservice v1beta1.metrics.k8s.io [root@k8s-master1:~]# kubectl top pods [root@k8s-master1:~]# kubectl logs -f -n kube-system metrics-server-69fbcb864b-jvkkr [root@k8s-master1:~]# cat metrics-server-deployment.yaml |
|
some other things to try: verify that 192.x.x.x IP addresses are in pod network; check kube-controller-manager process: "--cluster-cidr=" option which was configured during kubeadm init with "--pod-network-cidr" option verify that pod network was configured to use correct interface in each node (e.g. flanneld command has an "--iface option); I had to change my daemonset/kube-flannel-ds configuration to use the flannel (pod) network interface; otherwise it would default to an internal (VirtualBox) network interface check your kube-proxy process; if you see "--hostname-override=...", try removing this argument from the daemonset/kube-proxy yaml configuration try setting "- --enable-aggregator-routing=true" in kube-apiserver |
and others
Metric-server unable to resolve the hostname to scrape the metrics from kubelet.
I figured its not resolving the hostname from kubedns
and as mentioned in following issues: #105 (comment)
and #97
I did try to edit
kubectl -n kube-system edit deploy metrics-serverBut metrics-server pod entered the error state.The describe apiservice v1beta1.metrics.k8s.io have message:
10.101.248.96 being the clusterIP of the metric-server.
The text was updated successfully, but these errors were encountered: