Missing metrics on IKS (IBM Cloud) v1.13.12+IKS

[aliartiza75]

Hi,

I have deployed a managed k8s v1.13.12+IKS cluster on IBM Cloud but the issue is that when i try to get the metrics using the command given below, I get errors:

Metrics Server Configuration:

"name": "metrics-server",
            "image": "registry.au-syd.bluemix.net/armada-master/metrics-server-amd64:v0.3.3",
            "command": [
              "/metrics-server",
              "--metric-resolution=30s",
              "--tls-cert-file=/etc/metrics-server-certs/tls.crt",
              "--tls-private-key-file=/etc/metrics-server-certs/tls.key"
            ],

Command 1

$ kubectl top nodes

output:

error: metrics not available yet

Command 2

$ kubectl top pods

output:

W1217 15:23:21.703946   32431 top_pod.go:266] Metrics not available for pod default/elasticsearch-operator-sysctl-4lwgn, age: 1819h24m46.703940017s

error: Metrics not available for pod default/elasticsearch-operator-sysctl-4lwgn, age: 1819h24m46.703940017s

After looking around I found the following issues:

1. issue 1.

I want to know how to solve this issue because I don't want to change the configuration of metrics server manually(i don't have its manifests and don't to make it go to error state). Does someone know anything about this issue?

[serathius]

Hey, similarly to issue you linked: Log no metrics known for pod doesn't indicate the problem. Details: #349
This log is equivalent to 404 on HTTP server. Please provide other logs that include information about why scraping node failed.

[aliartiza75]

metrics-server logs:

I1217 11:11:59.043116       1 secure_serving.go:116] Serving securely on [::]:443
I1217 11:12:03.535874       1 log.go:172] http: TLS handshake error from X.X.X.X:58590: EOF
I1217 11:12:03.639408       1 log.go:172] http: TLS handshake error from X.X.X.X:38142: EOF
I1217 11:12:04.139413       1 log.go:172] http: TLS handshake error from X.X.X.X:39968: EOF
E1217 11:12:20.249261       1 reststorage.go:147] unable to fetch pod metrics for pod namespace1/xxx4: no metrics known for pod
E1217 11:12:20.255169       1 reststorage.go:147] unable to fetch pod metrics for pod namespace2/xxx3: no metrics known for pod
E1217 11:12:20.255178       1 reststorage.go:147] unable to fetch pod metrics for pod namespace1/xxx2 no metrics known for pod
E1217 11:12:20.255186       1 reststorage.go:147] unable to fetch pod metrics for pod namespace1/xxx1 no metrics known for pod

@serathius

[aliartiza75]

@serathius do let me know if you want something else from my side.

[serathius]

Log http: TLS handshake error from X.X.X.X:58590: EOF points to a problem with TLS configuration. Random ports numbers point to those being a client connections (when you call kubectl top node). This means that your apiserver has problem with accessing your metrics-server.
This could be verified by bypassing apiserver proxy and send curl requests to MetricsServer directly.

Overall it looks like your metrics-server has incorrect TLS configuration. If you cannot change metrics-server manifests, I don't think there is a way to fix it.

[aliartiza75]

@serathius this is a managed kubernetes cluster, so i don't think i can change anything because I don't want to break anything. :(

[aliartiza75]

@serathius kindly give your input

[serathius]

I don't think I we can do more without help of the cloud provider here. Metrics Server is part of Autoscaling pipeline which is not considered necessary for being certified by CNCF as Kubernetes compatible.

I think best way for you is to make the problem visible to your cloud provider by support tickets or forum post (don't now how it works on IBM cloud). I can provide documentation describing how k8s control plane needs to be configured for metrics server to work. Doc:

• https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
• https://kubernetes.io/docs/concepts/architecture/master-node-communication/#master-to-cluster

Only alternative would be for you to setup your own k8s cluster or try other vendors.

[aliartiza75]

after a little bit of investigation, I found out that the issue is with the IBM IKS cluster. Same thing works on other vendors.

[serathius]

Closing per Kubernetes issue triage policy

GitHub is not the right place for support requests.
If you're looking for help, check Stack Overflow and the troubleshooting guide.
You can also post your question on the Kubernetes Slack or the Discuss Kubernetes forum.
If the matter is security related, please disclose it privately via https://kubernetes.io/security/.