Turn automatic-recovery off by default

[himanshu-kun]

What would you like to be added:
MCM should by default turn the automatic recovery(https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-ec2-default-automatic-recovery/) off for an instance. automatic-recovery is a feature offered by AWS which will recover the instance on a new host in case of host failure, with the same instance id , volume attached.

Why is this needed:
Currently MCM itself has a health check mechanism where it terminates a machine in case its unhealthy(kubelet not responding or some other conditions) for over healthTimeout(by default 10 min). This means we have two health check actions which could race against each other.
If AWS autorecovery , happens before health-check , then its fine
But if it takes longer time (means the instance is still in transfer mode from one instance to other, volumes are detaching) then MCM recovery would kick in and it'll delete the instance on new host to start a new instance all together , leading to detachment of volumes again and a longer recovery which is undesirable.

[himanshu-kun]

Initial discussion could be found here https://sap-ti.slack.com/archives/CBVQLMS6N/p1659345327978849

It needs to be tested whether this kind of scenario could actually happen

[himanshu-kun]

Another question is :
Whether to keep the parameter configurable by the user. MCM health recovery is currently enough though, but there could be scenarios where customer would want to depend on the AWS recovery method

[himanshu-kun]

@D063648 do you still find this feature relevant ?

[rishabh-11]

Grooming Decision:-

Check if auto-recovery is progressing (Research is needed to see if this is possible). If yes, relook at the health timeout to allow the instance to recover.

[ashwani2k]

This may not be required if we introduce gardener/machine-controller-manager#755.
@rishabh-11 to take a call if we need to do this here or we can close this in favor of React faster if VM instance is gone (i.e. don’t wait until full machineHealthTimeout/machineCreationTimeout lapses) #755