Improve Cluster deletion

[cornelius-keller]

• Cluster deletion handle cleanup of cinder csi pv (will be a leftover, but don't block deletion)
• cluster deletion is broken when there exists a svc load balancer. (blocking deletion) .

[cornelius-keller]

@ericgraf This issue was created out of notes from you in team rocket inbox. Questions for refinement:

• do we need a web hook preventing cluster deletion in those cases?
• is this happening with capo deletion or is this an improvement to our cleanup scripts.

[cornelius-keller]

@cornelius-keller ask in kaas sync:

• if this is happening on other platforms as well.
• if we can have a generic web hook for capi clusters that checks if there is still pvs and svc type load balancer.

[ericgraf]

From KAAS.

• It is expected behaviour that we clean up all resources that we create ( ie pvcs, loadblanacers , etc... )
• This seems to be a capo specific issue. Azure does sometimes also have left over resources but doesn't block the deletion.
• If we do a finalizer to block deletion we should make sure it's openstack specific crs
• CAPO should be able to delete the cluster even it is blocked by resources it didn't create. ( cloud-provider-openstack creates the svc loadbalancer )
• A custom operator for CAPO might be required to handle this.

[erkanerol]

Related issue in upstream repo: kubernetes-sigs/cluster-api-provider-openstack#842

[erkanerol]

Solution	Requires working WC	Requires touching OpenStack	Extendability for other types
Cleaning OpenStack Resources	NO	YES	Depends on support in openstack tools
Cleaning Kubernetes Resources in WC	YES	NO	YES
Blocking cluster deletion	YES	NO	YES

Cleaning OpenStackResources:

• In CAPO+CPO, tagging resources with cluster name
• Adding a new operator which deletes OpenStack resources for a deleting cluster by using tags
• Add volumes metadata #710

Cleaning Kubernetes Resources in WC:

• Adding a new operator or a pre-delete helm hook with a k8s job
• The job can run in MC or in WC

Blocking Cluster Deletion:

• By using ValidatingAdmissionWebhook in MC

[erkanerol]

We are going to deploy a new operator which cleans loadbalancers&volumes created by users for the WC.

The repo of the operator: https://github.com/giantswarm/cluster-api-cleaner-openstack

[erkanerol]

The new operator is running in all OpenStack clusters.