Pivot fails: machines.cluster.k8s.io "controlplane-0" already exists

[vincepri]

/kind bug

What steps did you take and what happened:
Using the AWS provider, clusterctl create cluster ... with the usual workflow, after the controlplane-0 instance gets created, clusterctl fails with the error shown below.

I0311 15:12:54.136027   21097 clusterclient.go:763] Waiting for kubectl apply...
I0311 15:12:57.128700   21097 clusterdeployer.go:120] Pivoting Cluster API stack to target cluster
I0311 15:12:57.128735   21097 pivot.go:67] Applying Cluster API Provider Components to Target Cluster
I0311 15:12:57.128743   21097 clusterclient.go:763] Waiting for kubectl apply...
I0311 15:12:58.822860   21097 pivot.go:72] Pivoting Cluster API objects from bootstrap to target cluster.
I0311 15:12:58.822877   21097 pivot.go:83] Ensuring cluster v1alpha1 resources are available on the source cluster
I0311 15:12:58.822886   21097 clusterclient.go:792] Waiting for Cluster v1alpha resources to become available...
I0311 15:12:58.824601   21097 clusterclient.go:805] Waiting for Cluster v1alpha resources to be listable...
I0311 15:12:58.827280   21097 pivot.go:88] Ensuring cluster v1alpha1 resources are available on the target cluster
I0311 15:12:58.827293   21097 clusterclient.go:792] Waiting for Cluster v1alpha resources to become available...
I0311 15:12:58.933297   21097 clusterclient.go:805] Waiting for Cluster v1alpha resources to be listable...
I0311 15:12:58.972567   21097 pivot.go:93] Parsing list of cluster-api controllers from provider components
I0311 15:12:58.972589   21097 decoder.go:224] decoding stream as YAML
I0311 15:12:58.980490   21097 pivot.go:101] Scaling down controller aws-provider-system/aws-provider-controller-manager
I0311 15:12:58.989234   21097 pivot.go:101] Scaling down controller cluster-api-system/cluster-api-controller-manager
I0311 15:12:58.994608   21097 pivot.go:107] Retrieving list of MachineClasses to move
I0311 15:12:58.997547   21097 pivot.go:212] Preparing to copy MachineClasses: []
I0311 15:12:58.997572   21097 pivot.go:117] Retrieving list of Clusters to move
I0311 15:12:59.005549   21097 pivot.go:171] Preparing to move Clusters: [test1]
I0311 15:12:59.005564   21097 pivot.go:234] Moving Cluster default/test1
I0311 15:12:59.005568   21097 pivot.go:236] Ensuring namespace "default" exists on target cluster
I0311 15:12:59.154078   21097 pivot.go:247] Retrieving list of MachineDeployments to move for Cluster default/test1
I0311 15:12:59.157842   21097 pivot.go:287] Preparing to move MachineDeployments: []
I0311 15:12:59.157856   21097 pivot.go:256] Retrieving list of MachineSets not associated with a MachineDeployment to move for Cluster default/test1
I0311 15:12:59.160689   21097 pivot.go:331] Preparing to move MachineSets: []
I0311 15:12:59.160703   21097 pivot.go:265] Retrieving list of Machines not associated with a MachineSet to move for Cluster default/test1
I0311 15:12:59.163617   21097 pivot.go:374] Preparing to move Machines: [controlplane-0 controlplane-0]
I0311 15:12:59.163631   21097 pivot.go:385] Moving Machine default/controlplane-0
I0311 15:12:59.207910   21097 clusterclient.go:816] Waiting for Machine controlplane-0 to become ready...
I0311 15:12:59.247330   21097 pivot.go:399] Successfully moved Machine default/controlplane-0
I0311 15:12:59.247364   21097 pivot.go:385] Moving Machine default/controlplane-0
I0311 15:12:59.276213   21097 createbootstrapcluster.go:36] Cleaning up bootstrap cluster.
I0311 15:12:59.276225   21097 kind.go:57] Running: kind [delete cluster --name=clusterapi]
I0311 15:13:00.705564   21097 kind.go:60] Ran: kind [delete cluster --name=clusterapi] Output: Deleting cluster "clusterapi" ...
F0311 15:13:00.705589   21097 create_cluster.go:61] unable to pivot cluster api stack to target cluster: unable to pivot cluster API objects: Failed to move cluster: default/test1: failed to move Machine default:controlplane-0: error copying Machine default/controlplane-0 to target cluster: error creating a machine object in namespace default: machines.cluster.k8s.io "controlplane-0" already exists

What did you expect to happen:
Controlplane and cluster should be created successfully.

[vincepri]

/milestone v1alpha1
/assign @detiber
/priority important-critical

[vincepri]

Seems the main issue here is that controlplane-0 seems duplicated

I0311 15:12:59.163617   21097 pivot.go:374] Preparing to move Machines: [controlplane-0 controlplane-0]

Investigating.

[ashish-amarnath]

The problem is with https://github.com/kubernetes-sigs/cluster-api/blob/master/cmd/clusterctl/clusterdeployer/clusterclient/clusterclient.go#L383
The machines are double counted.

[ashish-amarnath]

/assign
/lifecycle active

[ashish-amarnath]

I could have saved myself a couple of hours had I come to this issue sooner.
I came here when I was ready to open an issue and a PR. 🐑

[leakingtapan]

I had the same issue. After this happened, I had a lingering cluster with the bootstrap kind cluster already deleted. How can I get rid of the half baked cluster at this point? Seems clusterctl delete cluster requires the kubeconfig for the created cluster. But where can I find it?

[detiber]

@leakingtapan The kubeconfig for the created cluster should be in the current working directory where you ran the clusterctl command and should be named kubeconfig.

@leakingtapan When you ran into this issue, where you running from a release or did you build from the master branch? If the later was it after the fix for this was already merged?

[leakingtapan]

@detiber I was using the release. Will check latest branch.

I tried deleting cluster using the saved kubeconfig. But the delete process is trying to pivot cluster back to bootstrap and it couldn't find it because the initial creation pivot is not success:

$ clusterctl  --kubeconfig kubeconfig delete cluster --provider-components aws/out/provider-components.yaml --bootstrap-type kind
I0318 10:45:42.025655   92032 clusterdeployer.go:149] Creating bootstrap cluster
I0318 10:45:42.025683   92032 createbootstrapcluster.go:27] Creating bootstrap cluster
I0318 10:46:44.926455   92032 clusterdeployer.go:157] Pivoting Cluster API stack to bootstrap cluster
I0318 10:46:44.926502   92032 pivot.go:67] Applying Cluster API Provider Components to Target Cluster
I0318 10:46:47.598696   92032 pivot.go:72] Pivoting Cluster API objects from bootstrap to target cluster.
I0318 11:03:07.643230   92032 createbootstrapcluster.go:36] Cleaning up bootstrap cluster.
F0318 11:03:10.476756   92032 delete_cluster.go:53] unable to pivot Cluster API stack to bootstrap cluster: unable to pivot cluster API objects: cluster v1alpha1 resource not ready
on source cluster

Is there anything im missing? I just started playing cluster-api, sorry if I asked dum questions