improve kubeadm's preflight and cluster health assurance

[neolit123]

with the recent failures that kubeadm exposed in cluster-api (see kubernetes-sigs/cluster-api#2769) next to retries and generic robustness there is something else we can improve.

as suggested by @timothysc we have the potential of extending the kubeadm assurance that a cluster is a good state using preflight or a tool such as the node-problem-detector (NPD) with the idea that a node should fail early instead of retrying everywhere in it's phases.

however, from my investigation some time ago the NPD was not very actively maintained.

we have some options that can be discussed:

• extend preflight on join with etcd and k8s client calls to make sure the etcd / api-server are healthy.
• post init optionally run performance / benchmarks on the control-plane.
• document that a tool like the NPD is recommended or even run it automatically.
• write a new tool and host it in the k/kubeadm repo that can be used for this and document it.

related issues about kubeadm join robustness and retries (that @fabriziopandini recently logged):
#2094
#2093
#2092
#2091
#2095

[ereslibre]

As we have discussed on the kubeadm office hours, the fact that this check passes does not ensure that later on during the kubeadm execution you won't have inestability.

Usually, kubeadm users will relay on an external load balancer, that is where kubeadm will point to, and we cannot control how this load balancer is set up or managed -- thus, we cannot control its behavior. So from the kubeadm side, we could try a number of requests N, effectively succeeding this test, and move forward; later on, the very next request N + 1 hits a different apiserver instance that is not ready to process our requests.

Based on our discussions, I think it would be better to prepare kubeadm for retries when it comes to requests issued to the apiserver, and ensuring that these requests are retried to some extent upon failure, rather than having a preflight check that we cannot exhaustively ensure won't fail afterwards.

That being said, an explicit preflight check can be added; it will check setups with obvious failures where the load balancer is not correctly set up at all, or where there are not apiservers at all, or when the load balancer decided to point us to a faulty apiserver on the preflight apiserver check request. But, it cannot be understood as an exhaustive health check in my opinion.

Also, as @fabriziopandini pointed out, the biggest source of inestability happens afterwards, when kubeadm is altering the system (so the load balancer detects a new apiserver started by kubeadm itself, impacting kubeadm's later own requests to the load balancer), or when modifying the etcd members.

[neolit123]

That being said, an explicit preflight check can be added; it will check setups with obvious failures where the load balancer is not correctly set up at all, or where there are not apiservers at all, or when the load balancer decided to point us to a faulty apiserver on the preflight apiserver check request. But, it cannot be understood as an exhaustive health check in my opinion.

i'm starting to see less value in such a preflight check.

on the other hand the node problem detector is a daemon-set that will not grant us much benefit for some of the failures we are trying to fix.

closing as per the comments during the planning session.
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

That being said, an explicit preflight check can be added; it will check setups with obvious failures where the load balancer is not correctly set up at all, or where there are not apiservers at all, or when the load balancer decided to point us to a faulty apiserver on the preflight apiserver check request. But, it cannot be understood as an exhaustive health check in my opinion.

i'm starting to see less value in such a preflight check.

on the other hand the node problem detector is a daemon-set that will not grant us much benefit for some of the failures we are trying to fix.

closing as per the comments during the planning session.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.