Failed first control plane node creates unrecoverable failure

[scottslowe]

What steps did you take and what happened:

I created a workload cluster on AWS that would use existing infrastructure. The instance backing the first control plane node failed unexpected (some sort of AWS failure), generating errors like this in the CAPA controller manager log:

I0424 16:51:13.725055       1 awsmachine_controller.go:361] controllers/AWSMachine "msg"="Error state detected, skipping reconciliation" "awsCluster"="capi-etcd" "awsMachine"="capi-etcd-control-plane-4jpj4" "cluster"="capi-etcd" "machine"="capi-etcd-control-plane-hmx8d" "namespace"="default"

In talking with @randomvariable, he suggested deleting the Machine object, which I did, and that removed the failed EC2 instance and created new Machine and AWSMachine objects, but further reconciliation never happened. Naadir indicated that this was "technically correct" behavior and that the KCP would never recover, but suggested raising this issue to possibly improve docs or surface errors better.

What did you expect to happen:

I expected the workload cluster to be created.

Anything else you would like to add:

N/A

Environment:

• Cluster-api version: 0.3.3
• Minikube/KIND version: N/A (tested on AWS)
• Kubernetes version: (use kubectl version): v1.16.4 (management), v1.17.3 (workload)
• OS (e.g. from /etc/os-release): Ubuntu 18.04

/kind bug

[detiber]

There is a unique challenge here. If we can differentiate between the control plane never initialized and the control plane existed but is no longer there, then I think we can safely retry initialization on failed creation.

I'm not sure we'd ever want to "re-initialize" a control plane that was present but is no longer there, since recreating the "control plane" despite the data loss incurred during the process would potentially cause more confusion than problems it would solve.

[scottslowe]

I'm completely out of my depth here, but wouldn't CABPK or KCP have knowledge as to whether the control plane ever initialized or not? I completely agree we'd want to avoid "re-initializing" a control plane.

[vincepri]

/priority awaiting-more-evidence
/milestone v0.3.x

[randomvariable]

CABPK is very much in the fire and forget category.
KCP potentially has the ability to figure this out. Related to #2976

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[vincepri]

/milestone v0.4.0

We should probably tackle this as part of KCP in a very narrow scenario (initialized)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[vincepri]

/milestone v0.4.x

[fabriziopandini]

/close

As of today the easiest path is to delete and re-create.

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close

As of today the easiest path is to delete and re-create.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.