MachineHealthCheck will fail until Cluster is Provisioned

[zawachte]

What steps did you take and what happened:

So I am applying a machine health check at the same time I apply the rest of the "cluster template" and I see in the controller logs that the get remote cluster client is always failing until the cluster is provisioned. Which makes sense, at this point, there is no kubeconfig secret.

What did you expect to happen:

This leads me to two questions...

1. Is it not recommended to apply a MHC before the cluster is provisioned?
2. If 1. is not the case ... should we just check cluster status before continuing with the MHC reconcile?

/kind bug

[vincepri]

Parts of this will be fixed and improved with @ncdc's work on #3752

[vincepri]

In general, the log failures are expected if the cluster isn't reachable or up yet. That said, we should do a better job to either ignore or delay these errors or logs whenever possible.

From a user perspective, one should be able to apply MHC whenever — the system should react based on the information it has about the cluster / machines that are being watched.

[zawachte]

#3752 looks like it improves all of this. thanks. Closing the issue.