Deleting the IBMPowerVSCluster object is not deleting the underneath IBMPowerVSImage

[mkumatag]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:

1. Deploy a Cluster, IBMPowerVSCluster and IBMPowerVSImage spec

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ibm-powervs-1
  name: ibm-powervs-1
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
        - 10.128.0.0/12
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: ibm-powervs-1-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: IBMPowerVSCluster
    name: ibm-powervs-1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: IBMPowerVSImage
metadata:
  name: ibmpowervsimage-sample
spec:
  clusterName: "ibm-powervs-1"
  serviceInstanceID: 97ff60d4-5b60-4a3d-bb28-34aedc603bf3
  bucket: "power-oss-bucket"
  object: "capibm-powervs-centos-treams8-1-22-4.ova.gz"
  region: "us-south"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: IBMPowerVSCluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ibm-powervs-1
  name: ibm-powervs-1
  namespace: default
spec:
  controlPlaneEndpoint:
    host: 158.175.162.5
    port: 6443
  network:
    name: mkumatag-capi-net
  serviceInstanceID: 97ff60d4-5b60-4a3d-bb28-34aedc603bf3
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: ibm-powervs-1-control-plane
  namespace: default
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
          - 192.168.151.5
          - 158.175.162.5
      controlPlaneEndpoint: 192.168.151.5:6443
      controllerManager:
        extraArgs:
          enable-hostpath-provisioner: "true"
    files:
      - content: |
          apiVersion: v1
          kind: Pod
          metadata:
            creationTimestamp: null
            name: kube-vip
            namespace: kube-system
          spec:
            containers:
            - args:
              - manager
              env:
              - name: vip_arp
                value: "true"
              - name: vip_interface
                value: "env2"
              - name: port
                value: "6443"
              - name: vip_cidr
                value: "29"
              - name: cp_enable
                value: "true"
              - name: cp_namespace
                value: kube-system
              - name: vip_ddns
                value: "false"
              - name: svc_enable
                value: "true"
              - name: vip_leaderelection
                value: "true"
              - name: vip_leaseduration
                value: "5"
              - name: vip_renewdeadline
                value: "3"
              - name: vip_retryperiod
                value: "1"
              - name: vip_address
                value: "192.168.151.5"
              image: docker.io/plndr/kube-vip:v0.3.7
              imagePullPolicy: Always
              name: kube-vip
              resources: {}
              securityContext:
                capabilities:
                  add:
                  - NET_ADMIN
                  - NET_RAW
                  - SYS_TIME
              volumeMounts:
              - mountPath: /etc/kubernetes/admin.conf
                name: kubeconfig
            hostNetwork: true
            volumes:
            - hostPath:
                path: /etc/kubernetes/admin.conf
              name: kubeconfig
          status: {}
        owner: root:root
        path: /etc/kubernetes/manifests/kube-vip.yaml
        permissions: "0744"
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
          eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
          provider-id: ibmpowervs://ibm-powervs-1/'{{ v1.local_hostname }}'
        name: '{{ v1.local_hostname }}'
    joinConfiguration:
      discovery:
        bootstrapToken:
          apiServerEndpoint: 192.168.151.5:6443
          caCertHashes: []
          token: ""
          unsafeSkipCAVerification: false
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          cloud-provider: external
          eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
          provider-id: ibmpowervs://ibm-powervs-1/'{{ v1.local_hostname }}'
        name: '{{ v1.local_hostname }}'
    preKubeadmCommands:
      - hostname "{{ v1.local_hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   {{ v1.local_hostname }}" >>/etc/hosts
      - echo "{{ v1.local_hostname }}" >/etc/hostname
    useExperimentalRetryJoin: true
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: IBMPowerVSMachineTemplate
      name: ibm-powervs-1-control-plane
  replicas: 1
  version: v1.22.5

2. Delete the Cluster spec

k delete cluster ibm-powervs-1

3. After step 1 and 2, IBMPowerVSImage is not getting deleted

What did you expect to happen:
Even IBMPowerVSImages should get deleted but it is not,

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[mkumatag]

IBMPowerVSCluster did trigger the delete action on the IBMPowerVSImage object but it is not really waiting for IBMPowerVSImage to be deleted, hence the child resource remain forever because in IBMPowerVSImage controller got a logic for checking for the cluster name and loops with that error like below:

[manager] E0203 13:52:00.244623      10 controller.go:317] controller/ibmpowervsimage "msg"="Reconciler error" "error"="failed to get Cluster/ibm-powervs-1: IBMPowerVSCluster.infrastructure.cluster.x-k8s.io \"ibm-powervs-1\" not found" "name"="ibmpowervsimage-sample" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="IBMPowerVSImage" 

you can see the deletionTimestamp is not none which means delete call did trigger on this resource

apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: IBMPowerVSImage
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"infrastructure.cluster.x-k8s.io/v1beta1","kind":"IBMPowerVSImage","metadata":{"annotations":{},"name":"ibmpowervsimage-sample","namespace":"default"},"spec":{"bucket":"power-oss-bucket","clusterName":"ibm-powervs-1","object":"capibm-powervs-centos-treams8-1-22-4.ova.gz","region":"us-south","serviceInstanceID":"97ff60d4-5b60-4a3d-bb28-34aedc603bf3"}}
    creationTimestamp: "2022-02-03T13:40:27Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2022-02-03T13:49:51Z"
    finalizers:
    - ibmpowervsimage.infrastructure.cluster.x-k8s.io
    generation: 2
    name: ibmpowervsimage-sample
    namespace: default
    ownerReferences:
    - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: IBMPowerVSCluster
      name: ibm-powervs-1
      uid: 5a7fae12-ad22-48ad-81ea-68e36acdbf23
    resourceVersion: "79449"
    uid: 9f5771ae-e4c0-488e-8b67-81bf216a0130
  spec:
    bucket: power-oss-bucket
    clusterName: ibm-powervs-1
    object: capibm-powervs-centos-treams8-1-22-4.ova.gz
    region: us-south
    serviceInstanceID: 97ff60d4-5b60-4a3d-bb28-34aedc603bf3
    storageType: tier1
  status:
    conditions:
    - lastTransitionTime: "2022-02-03T13:43:47Z"
      message: imageImport operation failed::download of image file capibm-powervs-centos-treams8-1-22-4.ova.gz
        from power-oss-bucket has failed, error:download of file power-oss-bucket/capibm-powervs-centos-treams8-1-22-4.ova.gz
        from COS region us-south failed, error:404 Not Found
      reason: ImageImportFailed
      severity: Error
      status: "False"
      type: ImageImported
    imageState: failed
    jobID: d51aeaff-b1ab-4d90-8518-0bcde88deb66
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[mkumatag]

capi cluster controller got a similar situation and they are handling via checking for the children and wait for them to delete and here is the logic, can be used in the IBMPowerVSCluster controller

https://github.com/kubernetes-sigs/cluster-api/blob/608340af81ba2025a9eb41891af9124919b3340c/internal/controllers/cluster/cluster_controller.go#L218:L260