CAPA WC deletion issue on goat / for private clusters

[AndiDog]

No description provided.

[AndiDog]

The issue seems to be that aws-network-topology-operator immediately detaches the TGW, resulting in lost connectivity from MC to WC (and also from laptop to WC). With this, the MachinePool deletion is already stuck since it requires writing node information back into etcd. We can either fix this with 1) #1827 since AWSCluster gets deleted last, or 2) wait until CAPI has fully deleted the Cluster which we are currently watching, i.e. delay reconciliation until the CAPI finalizer is gone (which however creates an implicit dependency on their finalizer name and on CAPI in general).

Relevant logs:

aws-network-topology-operator-68c768dd4b-6mnqk aws-network-topology-operator 2023-08-10T14:07:31.026108831+02:00 2023-08-10T12:07:31Z	INFO	transitgateway-registrar	TransitGateway detached from VPC	{"controller": "cluster", "controllerGroup": "cluster.x-k8s.io", "controllerKind": "Cluster", "cluster": {"name":"andreas86","namespace":"org-giantswarm"}, "namespace": "org-giantswarm", "name": "andreas86", "reconcileID": "3b7ee654-8702-4ba6-8ebf-7f26ba08873d", "vpcID": "vpc-0bb670cd59f66a634", "transitGatewayID": "tgw-019120b363d1e81e4"}

capi-controller-manager-67bcfcdd7-gtpvl manager 2023-08-10T14:17:11.112708997+02:00 E0810 12:17:11.112603       1 controller.go:329] "Reconciler error" err="failed to add &Node{ObjectMeta:{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{},Allocatable:ResourceList{},Phase:,Conditions:[]NodeCondition{},Addresses:[]NodeAddress{},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:0,},},NodeInfo:NodeSystemInfo{MachineID:,SystemUUID:,BootID:,KernelVersion:,OSImage:,ContainerRuntimeVersion:,KubeletVersion:,KubeProxyVersion:,OperatingSystem:,Architecture:,},Images:[]ContainerImage{},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} watch on cluster org-giantswarm/andreas86: failed to create cluster accessor: error creating dynamic rest mapper for remote cluster \"org-giantswarm/andreas86\": Get \"https://internal-andreas86-apiserver-1467918663.eu-north-1.elb.amazonaws.com:6443/api?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="org-giantswarm/andreas86-48nqb" namespace="org-giantswarm" name="andreas86-48nqb" reconcileID=7a95118b-b971-46ac-8320-8907af27a272

[mnitchev]

Done in https://github.com/giantswarm/aws-resolver-rules-operator/releases/tag/v0.13.0