"cannot set blockOwnerDeletion" when using OpenShift as management cluster

[mdbooth]

What steps did you take and what happened:
Attempted to create a cluster using clusterctl v0.4.0 and infrastructure openstack:v0.4.0-beta.0 using OpenShift 4.9 as the management cluster.

capi-controller-manager fails with:

E0706 10:51:34.897458       1 controller.go:302] controller-runtime/manager/controller/cluster "msg"="Reconciler error" "error"="[openstackclusters.infrastructure.cluster.x-k8s.io \"capi-quickstart\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io \"capi-quickstart-control-plane\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>]" "name"="capi-quickstart" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"

What did you expect to happen:
Success.

Anything else you would like to add:
This appears to be the same as #3274, which was using AWS in place of OpenStack. It wouldn't appear to be a CAPO issue.

Environment:

• Cluster-api version:

$ clusterctl version
clusterctl version: &version.Info{Major:"0", Minor:"4", GitVersion:"v0.4.0", GitCommit:"7f879be68d15737e335b6cb39d380d1d163e06e6", GitTreeState:"clean", BuildDate:"2021-06-24T21:53:42Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

• Minikube/KIND version:
  N/A (OpenShift 4.9 Nightly)

• Kubernetes version: (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b3b92b2", GitCommit:"b3b92b2", GitTreeState:"clean", BuildDate:"2021-01-27T21:38:11Z", GoVersion:"go1.16beta1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+0228142", GitCommit:"0228142e0871232fac461da3c448510643ede06f", GitTreeState:"clean", BuildDate:"2021-07-02T20:58:04Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
  RHCOS

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[mdbooth]

This appears to relate to https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement

It affects other objects:

E0706 12:49:56.666143       1 machinedeployment_sync.go:219] controller-runtime/manager/controller/machinedeployment "msg"="Failed to create new machine set" "error"="machinesets.cluster.x-k8s.io \"capi-quickstart-md-0-5fcccfbd4b\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>" "name"="capi-quickstart-md-0" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "machineset"="capi-quickstart-md-0-5fcccfbd4b"
E0706 12:49:56.666252       1 machinedeployment_controller.go:145] controller-runtime/manager/controller/machinedeployment "msg"="Failed to reconcile MachineDeployment" "error"="machinesets.cluster.x-k8s.io \"capi-quickstart-md-0-5fcccfbd4b\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>" "name"="capi-quickstart-md-0" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment"
E0706 12:49:56.666727       1 controller.go:302] controller-runtime/manager/controller/machinedeployment "msg"="Reconciler error" "error"="machinesets.cluster.x-k8s.io \"capi-quickstart-md-0-5fcccfbd4b\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>" "name"="capi-quickstart-md-0" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment"

[mdbooth]

I have worked round this problem by adding the finalizers resource to everything in cluster.x-k8s.io in ClusterRole capi-system-capi-manager-role.

i.e.

- apiGroups:
  - cluster.x-k8s.io
  resources:
  - clusters
  - clusters/status
  - clusters/finalizers

and the same for clusters, machinedeployments, machinehealthchecks, machinepools, machines, and machinesets.

[vincepri]

Do we know when the finalizer enforcement was introduced?

cc @fabriziopandini @randomvariable

/milestone v0.4

[mdbooth]

Do we know when the finalizer enforcement was introduced?

cc @fabriziopandini @randomvariable

/milestone v0.4

Is it possible that non-OpenShift clusters don't normally have OwnerReferencesPermissionEnforcement enabled, and OpenShift clusters aren't typically used as capi management clusters? i.e. It's latent, but sufficiently uncommon that you just don't see it often?

[vincepri]

We have e2e tests on all the latest versions of Kubernetes. If this feature needs to be enabled, we probably didn't catch it yet.

[randomvariable]

ooh, interesting, let me find out.

[randomvariable]

The default list of admission controllers is, as of v1.21:
NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota.

I do think this is because OpenShift turns on some additional ones by default, and it's caught out some controllers in k/k as recently as April.

Kubebuilder nowadays defaults to adding the finalizer permissions, we probably started before that was included in the templating. Makes sense to add the same annotations.

[randomvariable]

/lifecycle active
/assign