VPC cluster deletion stuck while deleting the subnet

[mkumatag]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
While deleting the vpc cluster it failed to list the vpc in the first pass and later it failed subsequently to fail the subnet because its already deleted and controller goes into infinite loop for deleting resources.

Initial vpc deletion failure because subnet was still present(took sometime to delete the subnet post delete call) while deleting the vpc

[manager] DELETE /v1/vpcs/r006-c47729a9-2277-47ba-9665-f9d036f4a3a2?generation=2&version=2022-09-13 HTTP/1.1
[manager] Host: us-south.iaas.cloud.ibm.com
[manager] User-Agent: vpc-go-sdk-0.29.0 (arch=arm64; os=linux; go.version=go1.19.4)
[manager] Authorization: [redacted]
[manager] Accept-Encoding: gzip
[manager] 
[manager] 
[manager] 2023/01/13 09:47:12 [Debug] Response:
[manager] HTTP/2.0 409 Conflict
[manager] Content-Length: 233
[manager] Cache-Control: max-age=0, no-cache, no-store, must-revalidate
[manager] Cf-Cache-Status: DYNAMIC
[manager] Cf-Ray: 788d35870e373c00-BLR
[manager] Content-Type: application/json; charset=utf-8
[manager] Date: Fri, 13 Jan 2023 09:47:12 GMT
[manager] Expires: -1
[manager] Pragma: no-cache
[manager] Server: cloudflare
[manager] Strict-Transport-Security: max-age=31536000; includeSubDomains
[manager] Vary: Accept-Encoding
[manager] X-Content-Type-Options: nosniff
[manager] X-Request-Id: 3e098e37-fec7-4854-951d-95a29258d5f8
[manager] X-Xss-Protection: 1; mode=block
[manager] 
[manager] {"errors":[{"code":"vpc_in_use","message":"Delete VPC failed: VPC still contains subnets","target":{"name":"id","type":"parameter","value":"r006-c47729a9-2277-47ba-9665-f9d036f4a3a2"}}],"trace":"3e098e37-fec7-4854-951d-95a29258d5f8"}
[manager] E0113 09:47:12.187890      11 controller.go:326]  "msg"="Reconciler error" "error"="failed to delete VPC: Delete VPC failed: VPC still contains subnets" "IBMVPCCluster"={"name":"ibm-vpc-0","namespace":"default"} "controller"="ibmvpccluster" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="IBMVPCCluster" "name"="ibm-vpc-0" "namespace"="default" "reconcileID"="202638c5-0961-4b54-875c-4e1a1dc2fc60"
[manager] I0113 09:47:12.188042      11 recorder.go:103] events "msg"="Failed vpc deletion - Delete VPC failed: VPC still contains subnets" "object"={"kind":"IBMVPCCluster","namespace":"default","name":"ibm-vpc-0","uid":"93018f1e-38e8-4a28-b9e8-3c6893db84aa","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"366183"} "reason"="FailedDeleteVPC" "type"="Warning"
[manager] I0113 09:47:12.188914      11 recorder.go:103] events "msg"="Failed vpc deletion - Delete VPC failed: VPC still contains subnets" "object"={"kind":"IBMVPCCluster","namespace":"default","name":"ibm-vpc-0","uid":"93018f1e-38e8-4a28-b9e8-3c6893db84aa","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"366183"} "reason"="FailedDeleteVPC" "type"="Warning"

try deleting the subnet after that and got stuck in that loop forever.

[manager] DELETE /v1/subnets/0717-2ea2b0d4-4783-4fd5-900b-245cdb040eee?generation=2&version=2022-09-13 HTTP/1.1
[manager] Host: us-south.iaas.cloud.ibm.com
[manager] User-Agent: vpc-go-sdk-0.29.0 (arch=arm64; os=linux; go.version=go1.19.4)
[manager] Authorization: [redacted]
[manager] Accept-Encoding: gzip
[manager] 
[manager] 
[manager] 2023/01/13 10:06:14 [Debug] Response:
[manager] HTTP/2.0 404 Not Found
[manager] Cache-Control: max-age=0, no-cache, no-store, must-revalidate
[manager] Cf-Cache-Status: DYNAMIC
[manager] Cf-Ray: 788d51687ad83c12-BLR
[manager] Content-Type: application/json; charset=utf-8
[manager] Date: Fri, 13 Jan 2023 10:06:14 GMT
[manager] Expires: -1
[manager] Pragma: no-cache
[manager] Server: cloudflare
[manager] Strict-Transport-Security: max-age=31536000; includeSubDomains
[manager] Vary: Accept-Encoding
[manager] X-Content-Type-Options: nosniff
[manager] X-Request-Id: e71f1876-5920-4f72-846c-40b91409f71f
[manager] X-Xss-Protection: 1; mode=block
[manager] 
[manager] {"errors":[{"code":"not_found","message":"Subnet not found","target":{"name":"id","type":"parameter","value":"0717-2ea2b0d4-4783-4fd5-900b-245cdb040eee"}}],"trace":"e71f1876-5920-4f72-846c-40b91409f71f"}
[manager] I0113 10:06:14.540261      11 recorder.go:103] events "msg"="Failed subnet deletion - Subnet not found" "object"={"kind":"IBMVPCCluster","namespace":"default","name":"ibm-vpc-0","uid":"93018f1e-38e8-4a28-b9e8-3c6893db84aa","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"366183"} "reason"="FailedDeleteSubnet" "type"="Warning"
[manager] E0113 10:06:14.540874      11 controller.go:326]  "msg"="Reconciler error" "error"="failed to delete subnet: Error when deleting subnet : Subnet not found" "IBMVPCCluster"={"name":"ibm-vpc-0","namespace":"default"} "controller"="ibmvpccluster" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="IBMVPCCluster" "name"="ibm-vpc-0" "namespace"="default" "reconcileID"="d0abacfc-1f62-42af-855d-4b12ea7afb4e"
[manager] I0113 10:06:14.541046      11 recorder.go:103] events "msg"="Failed subnet deletion - Subnet not found" "object"={"kind":"IBMVPCCluster","namespace":"default","name":"ibm-vpc-0","uid":"93018f1e-38e8-4a28-b9e8-3c6893db84aa","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"366183"} "reason"="FailedDeleteSubnet" "type"="Warning"

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):