Add phases support to clusterctl

[detiber]

Currently the clusterctl workflows are pretty monolithic and more advanced usage (such as using existing clusters for bootstrapping, or avoiding pivoting the cluster-api components) has required adding additional flags. It would help if we exposed logical phases to users to allow them to create more advanced workflows without buying into the existing clusterctl workflows or having to leverage complex interactions between command line flags.

In order to avoid an incredibly large PR that introduces full phase support, I propose that we start breaking out existing logical functionality, re-using it from the existing monolithic workflows, exposing it as a phase, and then once there is a good set of logical phases the existing subcommands can be modified to perform the required phases step by step.

Per the conversation in the weekly cluster-api meeting on 24 Oct, 2018, these phases should be exposed under an alpha subcommand to avoid user expectations of long term support for the currently defined "phases".

Proposed tasks (I suspect I may be missing a few in my initial attempt to break it down):

• Add create-bootstrap-cluster phase [clusterctl] add phases create-bootstrap-cluster subcommand #553
• Add apply-cluster-api-components phase Create a phase for applying the Cluster API components #573
• Add apply-addons phase Create a phase for applying the Addon components #577
• Add apply-cluster phase [clusterctl] add phases for apply-machines and apply-cluster #581
• Add apply-machines phase [clusterctl] add phases for apply-machines and apply-cluster #581
• Add phase for retrieving kubeconfig clusterctl - Add phase for get-kubeconfig #624
• Add pivot phase (To pivot the provider components, clusters, and machine* objects from a source cluster to a target cluster) Add pivot phase to clusterctl and unify pivoting approach #731
• Update create cluster command to use phases
• Update delete cluster command to use phases
• Update validate cluster command to use phases
• Update existing subcommands to use phases directly

The pivot phase mentioned above would require some changes to the existing workflow. Currently handling of pivoting is handled differently between create and delete actions. I would propose that the pivot should be modified to perform the following actions to create a more unified experience:

1. Create provider components in target cluster
2. Query and filter for the controller StatefulSets on source cluster (exact method for doing this across providers is tbd)
3. Scale the controller StatefulSets to 0 replicas on source cluster
4. For each cluster (across namespaces)
   1. copy cluster object to target cluster/namespace
   2. copy machine* objects to target namespace
   3. force delete machine* and cluster objects from source cluster/namespace (method for doing this without triggering finalizers tbd)
5. Delete providercomponents from source cluster

[detiber]

/assign

[roberthbailey]

I've added this to the weekly meeting topics for discussion.

[ingvagabund]

I am exercising a slightly different workflow when provisioning a k8s cluster with cluster API stack deployed. Basically trying to skip the requirement for bootstrapping cluster. Instead, I am running an actuator directly to provision a single master node and doing all the work there. In steps:

1. call actuator directly to provision a master machine (machine provider config points to user data that run kubeadm init).
2. get kubeconfig and master DNS public name
3. deploy the cluster API stack in the cluster made by the master node
4. provision a worker/compute node via machineset deployed in the cluster made by the master node

There is no pivoting so it's very simplified from the original workflow the clusterctl currently supports. Still, there are some phases that overlap:

• apply-cluster-api-stack
• apply-addons
• create-nodes

I consider this simplification useful since (e.g. in the case of the aws actuator), I only need a single binary that I execute and get a running cluster in AWS without existing cluster.

So it's very demanding to have phases I can run in different order than the default sequence.

[roberthbailey]

@ingvagabund - for your workflow, how are you are running the cluster api stack on the master? Are you using labels / taints?

[ingvagabund]

Are you using labels / taints?

I deploy the same manifests as generated by the kustomize under [1]. So the taints and tolerations. Are there any other recommended requirements other than run kubeadm init and deploy all the manifests on top of that?

[1] https://github.com/kubernetes-sigs/cluster-api/tree/master/config

EDIT: node tainting can be done as part of user data script. Not saying it's the best way but sufficient for development purposes.

[detiber]

/lifecycle active

[ncdc]

/milestone Next

[vincepri]

/area clusterctl

[vincepri]

@detiber Is this one still relevant? Should we punt to Next?

[detiber]

@vincepri closing, since clusterctl has a limited shelf life anyway.
/close

[k8s-ci-robot]

@detiber: Closing this issue.

In response to this:

@vincepri closing, since clusterctl has a limited shelf life anyway.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.