unable to fully collect metrics #130
Labels
kind/support
Categorizes issue or PR as a support question.
Comments
|
@vvkkhjt try this flag --kubelet-insecure-tls |
|
the |
|
@MIBc when i use that flag: streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=15, ErrCode=NO_ERROR, debug=""
E0906 06:28:39.309417 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=15, ErrCode=NO_ERROR, debug=""
E0906 06:28:39.312965 1 reflector.go:322] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Failed to watch *v1.Pod: Get https://10.254.0.1:443/api/v1/pods?resourceVersion=856148&timeoutSeconds=395&watch=true: dial tcp 10.254.0.1:443: connect: connection refused
E0906 06:28:39.313303 1 reflector.go:322] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Failed to watch *v1.Node: Get https://10.254.0.1:443/api/v1/nodes?resourceVersion=858077&timeoutSeconds=440&watch=true: dial tcp 10.254.0.1:443: connect: connection refused |
That implies that metrics-server isn't able to contact your main kubernetes API server. |
|
@DirectXMan12 but the other services are working good。。when i use helm install heapster,it works good either。 helm install stable/heapster --name heapster --namespace kube-system --set "rbac.create=true,command[0]=/heapster,command[1]=--source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true" --debugthen k top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.168.220.50 254m 3% 3606Mi 11%
192.168.66.64 226m 5% 2096Mi 57%
192.168.66.66 99m 2% 1388Mi 62% |
|
Hi all, k8s: v1.11 I am not using TLS, However, I fall into similar problem(not being able to After receiving the kubelet's sumary(json) there is such a line: Another message says it cannot parse some init containers. Is this the reason it is failing? Best! E0924 12:45:08.398855 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s1: unable to get a valid timestamp for metric point for container "install-cni" in pod kube-system/kube-router-2l6f5 on node "10.36.10.1", discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape metrics from source kubelet_summary:k8s4: unable to get a valid timestamp for metric point for container "install-cni" in pod kube-system/kube-router-6zvxd on node "10.36.10.4", discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape |
|
@brunovlucena what happens when you run |
|
i have the same problem |
Error from server (NotFound): nodemetrics.metrics.k8s.io "k8s1" not found |
|
is there only one node? |
|
|
|
That |
|
@DirectXMan12 I have tried the I have also tried to add that on the line where the source is defined, like this: I've tried with both v1.4.0 and v1.5.4 of heapster. Is there a way to tell heapster to accept the cert without that IP SAN setting? Thank you, |
|
i have the same problem... |
|
@lbudai74 this is the repository for metrics-server, not heapster. Heapster's repository can be found at kubernetes/heapster. |
|
@tengzhexiao which problem? |
|
I am also facing the same issue |
|
@Rajat-0 I think you're facing a different issue. If your kubelet lists the preferred address as 127.0.0.1, metrics-server can't collect from it, because it's the 127.0.0.1 is the loopback adapter (i.e. localhost in the pod is not localhost in the node). |
|
i have the same problem |
|
what about a feature that has watches a namespace for a configmap per node that contains the ca to use to verify that node? Nodes can self register self signed ca's then. |
|
well I have the same issue, opened a ticket here: #214 tried to be as much as informative as possible. in v0.3.1 it ignores completely from pods with init containers. and why is that ? because init containers returns empty stats in stats summary api call (unless you are using docker on the kubernetes nodes). |
|
@DirectXMan12 Any ideas would be appreciated. |
when i use kubectl top node ,I get the same error like you ,have you fix it ? |
|
@Kingchan230 Yes I resolved my issue. In my case the problem was that I applied some resources from If something is unclear feel free to reach me back. |
|
i have the same problem |
|
I had the same issue and re-applying the Prometheus adapter API service file fixed it: |
|
FIxed for me on EKS |
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
@DirectXMan12 I use The result shows that I got CPU, memory and other information of all nodes, but only node47 node is displayed. |
|
watch this：#278 |
serathius
added
kind/support
|
Closing per Kubernetes issue triage policy GitHub is not the right place for support requests. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
use k8s v1.11.2 && metrics-server v0.3.0
Did the following:
KUBE_API_ARGS="--authorization-mode=Node,RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kube-proxy.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-proxy-key.pem"KUBELET_ARGS="--network-plugin=cni --cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --authentication-token-webhook=true --read-only-port=10255"then metrics-server logs show this:
unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.66.66: unable to fetch metrics from Kubelet 192.168.66.66 (192.168.66.66): Get https://192.168.66.66:10250/stats/summary/: x509: cannot validate certificate for 192.168.66.66 because it doesn't contain any IP SANs, unable to fully scrape metrics from source kubelet_summary:192.168.66.64: unable to fetch metrics from Kubelet 192.168.66.64 (192.168.66.64): Get https://192.168.66.64:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.220.50: unable to fetch metrics from Kubelet 192.168.220.50 (192.168.220.50): Get https://192.168.220.50:10250/stats/summary/: x509: cannot validate certificate for 192.168.220.50 because it doesn't contain any IP SANs]when I use metrics-server v0.2.1,add metrcs-server args like this:
It seems to work,but when i use v0.3.0 and add args --source it's says
Error: unknown flag: --source,So I used the help instruction to find the corresponding parameter, but I didn't find it. So, how can I fix this problem?
This is my first time to mention issue, which is not so good. Sorry.
The text was updated successfully, but these errors were encountered: