Defer template deletion for MachineSets in a managed topologies

[fabriziopandini]

What steps did you take and what happened:
While testing template rotation for a MachineDeployment in a managed topology, deletion process of the old MachineSet got stuck and the MachineSet controller started to fail due to missing templates.

What did you expect to happen:
The machine deployment in a managed topology to properly rotate templates

Anything else you would like to add:
This is a problem of syncronization between the managed topology controller and the MachineSet controller.

Managed topology controller is responsible to delete old templates in case of template rotation, but this operation has to wait for the MachineSet to actually delete all the machines before removing the template.

The current working assumption to address this issue is to implement a MachineSet topology controller watching for MachineSets with the cluster.x-k8s.io/topology label only; this controller is going to:

• add a finaliser on the machine set if missing
• detect the MachineSet being deleted after removing all the machines
• delete the corresponding templates if rotated (if different from the ones in the MachineDeployment)
• remove the finaliser

Environment:

• Cluster-api version: master

/kind bug

[fabriziopandini]

/area topology
/milestone v0.4

[sbueringer]

@fabriziopandini Do you think we should also check that no other MachineSet is using the templates and only delete them if the "current" MachineSet is the last one? (not sure if we can get that 100% race condition free)

I think otherwise we run into problems if two MachineSets are using the same templates and both are in deletion.

[sbueringer]

Do we need a similar mechanism for MachineDeployments? (or maybe use the one for MachineSets for both)

[fabriziopandini]

We probably need both, but let's start by MachineSet because they have more frequent turnover than MachineDeployments

[sbueringer]

/assign

[fabriziopandini]

@sbueringer should we close this issue now that #5191 is merged?

[sbueringer]

Yup, let's close it.
/close

[k8s-ci-robot]

@sbueringer: Closing this issue.

In response to this:

Yup, let's close it.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.