Improve logic of VM-Deletion and Safey-controller for Azure

[hardikdr]

We need to consider the possibility of cloud-provider taking too long while deleting the VM. At the moment, we issue Delete() call for the VM, and if there is no immediate error-response we try replacing the VM in parallel. We have seen incidents[in Azure] where cloud-provider deletion of the VM was stuck on cloud-provider and which led MCM to create more machines than expected.
As an improvement we could enhance both deletion logic and safety-controller in following ways:

• Deletion Logic:

  • [Azure-specific]: Stop VM before deleting the VM. We understood from Microsoft contact that stopping the VM will pull out the VM from quota-calculation- this should also save some cost for us, as VM is not costed after stopping. Another supporting pointer is that deletion mostly gets stuck on Azure due to detaching/attaching of the VM, which will be avoided if stopped first- and deletion of the VM will anyway dependent on Azure which we anyways cannot do anything about.

• Safety-controller:

  • Currently, the freeze logic in safety-controller only checks the number of Machine objects and decides to freeze if they exceed the intended number.
  • We need to think about enabling similar logic for resources in cloud-provider. Essentially safety-controller should also consider VMs stuck in Deletion or Failed after deletion -- as active resources and freeze if they exceed the intended number. We could expose new configuration-knob for this number.

[chgeuer]

Hi @hardikdr one quick feedback on the deletion logic: I think you're operating under a wrong assumption. Here's the situation on Azure: When a VM is in status=Stopped (deallocated), it actually still counts against the vCPU core quota. In my sub, I have a single (stopped/de-allocated) 2-core VM, and my quota usage is 2/100...

The stopped/deallocated VM does not cost you by-the-minute CPU costs (because doesn't block cores on a physical host). Let's discuss that next Tuesday as well.

[prashanth26]

Closing this issue in favor of #242