clusterctl init on capz cluster fails

[CecileRobertMichon]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Use kind cluster to deploy a capz cluster and try to pivot according the instructions at https://cluster-api.sigs.k8s.io/clusterctl/commands/move.html#bootstrap--pivot

The same doesn't repro in an Azure k8s clusters not built by capz (eg. AKS cluster).

Related to kubernetes-sigs/cluster-api#2731

Seeing init getting stuck at Waiting for cert-manager to be available... consistently, even though cert-manager seems to be ready:

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes                                                                                                                     

NAME                         STATUS   ROLES    AGE   VERSION
capi-2-control-plane-5nws7   Ready    master   16m   v1.17.3
capi-2-control-plane-g7k5k   Ready    master   15m   v1.17.3
capi-2-control-plane-ps8nw   Ready    master   18m   v1.17.3
capi-2-md-0-ngw2x            Ready    <none>   17m   v1.17.3
capi-2-md-0-nwzh4            Ready    <none>   16m   v1.17.3
capi-2-md-0-qsb42            Ready    <none>   17m   v1.17.3

clusterctl --kubeconfig=./capi-quickstart.kubeconfig init --v 5                                                                                                                 

Installing the clusterctl inventory CRD
Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
Fetching providers
Fetching File="core-components.yaml" Provider="cluster-api" Version="v0.3.3"
Fetching File="bootstrap-components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.3"
Fetching File="control-plane-components.yaml" Provider="control-plane-kubeadm" Version="v0.3.3"
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.3"
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.3"
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.3"
Installing cert-manager
Creating Namespace="cert-manager"
Creating CustomResourceDefinition="challenges.acme.cert-manager.io"
Creating CustomResourceDefinition="orders.acme.cert-manager.io"
Creating CustomResourceDefinition="certificaterequests.cert-manager.io"
Creating CustomResourceDefinition="certificates.cert-manager.io"
Creating CustomResourceDefinition="clusterissuers.cert-manager.io"
Creating CustomResourceDefinition="issuers.cert-manager.io"
Creating ServiceAccount="cert-manager-cainjector" Namespace="cert-manager"
Creating ServiceAccount="cert-manager" Namespace="cert-manager"
Creating ServiceAccount="cert-manager-webhook" Namespace="cert-manager"
Creating ClusterRole="cert-manager-cainjector"
Creating ClusterRoleBinding="cert-manager-cainjector"
Creating Role="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating ClusterRoleBinding="cert-manager-webhook:auth-delegator"
Creating RoleBinding="cert-manager-webhook:webhook-authentication-reader" Namespace="kube-system"
Creating ClusterRole="cert-manager-webhook:webhook-requester"
Creating Role="cert-manager:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager:leaderelection" Namespace="kube-system"
Creating ClusterRole="cert-manager-controller-issuers"
Creating ClusterRole="cert-manager-controller-clusterissuers"
Creating ClusterRole="cert-manager-controller-certificates"
Creating ClusterRole="cert-manager-controller-orders"
Creating ClusterRole="cert-manager-controller-challenges"
Creating ClusterRole="cert-manager-controller-ingress-shim"
Creating ClusterRoleBinding="cert-manager-leaderelection"
Creating ClusterRoleBinding="cert-manager-controller-issuers"
Creating ClusterRoleBinding="cert-manager-controller-clusterissuers"
Creating ClusterRoleBinding="cert-manager-controller-certificates"
Creating ClusterRoleBinding="cert-manager-controller-orders"
Creating ClusterRoleBinding="cert-manager-controller-challenges"
Creating ClusterRoleBinding="cert-manager-controller-ingress-shim"
Creating ClusterRole="cert-manager-view"
Creating ClusterRole="cert-manager-edit"
Creating Service="cert-manager" Namespace="cert-manager"
Creating Service="cert-manager-webhook" Namespace="cert-manager"
Creating Deployment="cert-manager-cainjector" Namespace="cert-manager"
Creating Deployment="cert-manager" Namespace="cert-manager"
Creating Deployment="cert-manager-webhook" Namespace="cert-manager"
Creating APIService="v1beta1.webhook.cert-manager.io"
Creating MutatingWebhookConfiguration="cert-manager-webhook"
Creating ValidatingWebhookConfiguration="cert-manager-webhook"
Waiting for cert-manager to be available...

kubectl --kubeconfig=./capi-quickstart.kubeconfig get pods --all-namespaces                                                                                                     

NAMESPACE      NAME                                                 READY   STATUS    RESTARTS   AGE
cert-manager   cert-manager-69b4f77ffc-psxzj                        1/1     Running   0          3m27s
cert-manager   cert-manager-cainjector-576978ffc8-w2nr5             1/1     Running   0          3m27s
cert-manager   cert-manager-webhook-c67fbc858-c4v5b                 1/1     Running   1          3m26s
kube-system    calico-kube-controllers-576dfc659c-gftt4             1/1     Running   1          5m20s
kube-system    calico-node-4zbv6                                    1/1     Running   1          5m22s
kube-system    calico-node-bhpnf                                    1/1     Running   0          5m22s
kube-system    calico-node-c49b5                                    1/1     Running   1          5m22s
kube-system    calico-node-d8wwd                                    1/1     Running   0          5m22s
kube-system    calico-node-knvmh                                    1/1     Running   0          5m22s
kube-system    calico-node-mcz8b                                    1/1     Running   1          5m22s
kube-system    coredns-6955765f44-w2zhr                             1/1     Running   0          17m
kube-system    coredns-6955765f44-xgznx                             1/1     Running   0          17m
kube-system    etcd-capi-2-control-plane-5nws7                      1/1     Running   0          16m
kube-system    etcd-capi-2-control-plane-g7k5k                      1/1     Running   0          14m
kube-system    etcd-capi-2-control-plane-ps8nw                      1/1     Running   0          17m
kube-system    kube-apiserver-capi-2-control-plane-5nws7            1/1     Running   0          16m
kube-system    kube-apiserver-capi-2-control-plane-g7k5k            1/1     Running   0          14m
kube-system    kube-apiserver-capi-2-control-plane-ps8nw            1/1     Running   0          17m
kube-system    kube-controller-manager-capi-2-control-plane-5nws7   1/1     Running   1          16m
kube-system    kube-controller-manager-capi-2-control-plane-g7k5k   1/1     Running   0          14m
kube-system    kube-controller-manager-capi-2-control-plane-ps8nw   1/1     Running   1          17m
kube-system    kube-proxy-4ptdx                                     1/1     Running   0          16m
kube-system    kube-proxy-ctbh8                                     1/1     Running   0          17m
kube-system    kube-proxy-jd7f5                                     1/1     Running   0          16m
kube-system    kube-proxy-jv8mg                                     1/1     Running   0          16m
kube-system    kube-proxy-p72tw                                     1/1     Running   0          14m
kube-system    kube-proxy-v45gf                                     1/1     Running   0          16m
kube-system    kube-scheduler-capi-2-control-plane-5nws7            1/1     Running   1          16m
kube-system    kube-scheduler-capi-2-control-plane-g7k5k            1/1     Running   1          14m
kube-system    kube-scheduler-capi-2-control-plane-ps8nw            1/1     Running   1          17m

kubectl --kubeconfig=./capi-quickstart.kubeconfig get deploy -n cert-manager  
                                                                                                  
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           7m24s
cert-manager-cainjector   1/1     1            1           7m24s
cert-manager-webhook      1/1     1            1           7m23s

After 10 minutes, init fails with Error: timed out waiting for the condition.

EDIT: I see this when I describe the apiservice:

Message:               failing or missing response from https://10.100.0.172:443/apis/webhook.cert-manager.io/v1beta1: Get https://10.100.0.172:443/apis/webhook.cert-manager.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[nader-ziada]

I can confirm that I had the same issue on azure, but the same scenario worked fine on aws