metrics-server deployments cannot connect to kubelet InternalIP

[logan2211]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT / FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:27:49Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): v1.17.2
• Cloud provider or hardware configuration: bare-metal onprem
• OS (e.g. from /etc/os-release): Ubuntu Bionic
• Kernel (e.g. uname -a): 4.15.0-76-generic

What happened?

In my environment, node hostnames are not resolvable, so communication must be done over InternalIP endpoints.

I installed metrics-server, modifying the deployment args to include --kubelet-preferred-address-types=InternalIP. metrics-server cannot collect metrics from the cluster's kubelets because of the error unable to fetch metrics from Kubelet node1 (10.0.23.26): Get https://10.0.23.26:10250/stats/summary?only_cpu_and_memory=true: x509: cannot validate certificate for 10.0.23.26 because it doesn't contain any IP SANs]

I verified that the kubelet certs only include the hostname CN/SAN:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = node1-ca@1580686293
        Validity
            Not Before: Feb  2 22:31:33 2020 GMT
            Not After : Feb  1 22:31:33 2021 GMT
        Subject: CN = node1@1580686293
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:node1
    Signature Algorithm: sha256WithRSAEncryption
    ...

How to reproduce it (as minimally and precisely as possible)?

Deploy a cluster with kubeadm, then deploy metrics-server with the metrics-server-deployment arg --kubelet-preferred-address-types=InternalIP added. In order to see the IP SAN error shown above, the arg --v=3 must also be added, as the metrics-server default logging settings will not display it.

Anything else we need to know?

kubernetes-sigs/metrics-server#196 - same issue, but does not specify whether kubeadm was used to deploy the cluster.

There are numerous other issues outlining similar issues with metrics-server connecting to kubelet, some of which appear to be IP SAN related, ie many people who changed the preferred address types to InternalIP, and then had to add --kubelet-insecure-tls to get metrics-server working. Many commenters don't directly call out the IP SAN issue, probably because metrics-server logging settings do not display it by default.

kubernetes-sigs/metrics-server#131
specifically metrics-server is suggesting that the node should contain an IP SAN in this situation:
kubernetes-sigs/metrics-server#131 (comment)

[neolit123]

/close

hi, this is probably caused by the fact that the kubelet that kubeadm starts is self-signing it's serving certificate. this issue outlines the problem better:
#1223 (comment)
has a guide of how to manually sign.

this is the default kubelet behavior.
if you want to enable this functionality today, you must sign your own serving certificate for the kubelet.

there are also a workaround with a proxy, but i'm not sure how it's done exactly.
we have a tracking issue to document this, when we can.
#1602

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

/close

hi, this is probably caused by the fact that the kubelet that kubeadm starts is self-signing it's serving certificate. this issue outlines the problem better:
#1223 (comment)
has a guide of how to manually sign.

this is the default kubelet behavior.
if you want to enable this functionality today, you must sign your own serving certificate for the kubelet.

there are also some workarounds with a proxy, but i'm not sure how it's done exactly.
we have a tracking issue to document this.
#1602

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

this workaround i do not recommend:
#1223 (comment)
but it depends on how you've setup your metrics server.

[pkpfr]

Set the arg --kubelet-insecure-tls on the metrics server container.