add support for adopting external cluster #306

pdettori · 2024-12-09T16:54:49Z

Summary

Adds support for importing (adopting) external cluster as control plane

Related issue(s)

Fixes #163

Signed-off-by: Paolo Dettori <[email protected]>

cmd/kflex/common/cp.go

MikeSpreitzer · 2024-12-10T22:04:16Z

cmd/kflex/adopt/adopt.go

+	cp := common.GenerateControlPlane(c.Name, string(controlPlaneType), "", hook, hookVars)
+
+	util.PrintStatus(fmt.Sprintf("Adopting control plane %s of type %s ...", c.Name, controlPlaneType), done, &wg, chattyStatus)
+	if err := cl.Create(context.TODO(), cp, &client.CreateOptions{}); err != nil {


FYI, all writes should supply a FieldManager in the XXXOptions.

I prefer to leave this to a separate campaign/PR as there are many other places where there are writes so should probably be done consistently everywhere.

MikeSpreitzer · 2024-12-10T22:08:31Z

Yes, WIP. This still needs a doc update. I do not understand how the user passes the kubeconfig of the cluster to be adopted into the adopt subcommand. The adopt subcommand needs to get added into the command tree.

pdettori · 2024-12-11T02:54:57Z

Yes, WIP. This still needs a doc update. I do not understand how the user passes the kubeconfig of the cluster to be adopted into the adopt subcommand. The adopt subcommand needs to get added into the command tree.

@MikeSpreitzer - correct on all points - I started with the easiest updates, now working on the "adopt" subcommand and getting the kubeconfig. The current idea is that the user will supply the name of the context (and optionally the path of the Kubeconfig) in the Kubeconfig for the cluster to adopt as flags of the adopt subcommand, and the such subcommand will take care of creating ClusterRoles, ClusterRoleBindings, SA and Token in the adopted cluster so that the Token + cluster endpoint + CA cert can be used to build a new Kubeconfig for the adopted cluster to store in a secret in the namespace associated with the control plane in the hosting cluster.

Signed-off-by: Paolo Dettori <[email protected]>

francostellari · 2024-12-17T01:13:00Z

@pdettori I think you have some changes in chart/templates/operator.yaml that may be accidental

francostellari · 2024-12-17T01:15:13Z

docs/users.md

+To create a control plane of type `external` with the required options, run the command: 
+
+```shell
+kflex adopt --adopted-context <kubeconfig-context-of-external-cluster> cp5


Could a separated kubeconfig be passed with KUBECONFIG=... kflex... or `kflex --kubeconfig '?

if no value is passed to --adopted-context could it default to the current one... especially if one passes a separate kubeconfig with only one context in it?

As this command both create the secret and create the CP... it would be hard to use in combination with KS future Helm chart. The process would be

create/push the secret in the cluster manually or via kflex.

tell KS helm to create a CP using it (either upon installation or later on)

Could a separated kubeconfig be passed with KUBECONFIG=... kflex... or `kflex --kubeconfig '?

Yes. These are all the options for the adopt subcommand. You'd be looking at the --adopted-kubeconfig flag.

$ kflex adopt --help Adopt a control plane from an external cluster and switches the Kubeconfig context to the current instance Usage: kflex adopt <name> [flags] Flags: -c, --adopted-context string path to adopted cluster context in adopted kubeconfig -a, --adopted-kubeconfig string path to adopted cluster kubeconfig file. If unspecified, it uses the default Kubeconfig -s, --chatty-status chatty status indicator (default true) -x, --expiration-seconds int adopted token expiration in seconds. Default is one year. (default 31536000) -h, --help help for adopt -k, --kubeconfig string path to kubeconfig file -p, --postcreate-hook string name of post create hook to run -e, --set stringArray set post create hook variables, in the form name=value -d, --skip-url-override skip URL override, used for local debugging only -u, --url-override https://127.0.0.1 URL overrride for adopted cluster. Required when cluster address uses local host address, e.g. https://127.0.0.1 -v, --verbosity int log level

if no value is passed to --adopted-context could it default to the current one... especially if one passes a separate kubeconfig with only one context in it?

Currently you do need to pass the context. It might be confusing in the common developer scenario where there is one kubeconfig with a context for each cluster, since the "current" context is typically the kubeflex hosting cluster context. It might be possible when a different Kubeconfig is used (which needs to be specified with --adopted-kubeconfig) to check if there is only one context and in that case not require the use of --adopted-context. But I would leave this as a future feature/PR if we really think it is useful.

As this command both create the secret and create the CP... it would be hard to use in combination with KS future Helm chart...

One possible approach for using kflex to create bootstrap secret and the helm chart to create the CP would be to add an option to the adopt subcommand --create-bootstrap-secret-only - would that help?

As this command both create the secret and create the CP... it would be hard to use in combination with KS future Helm chart...

One possible approach for using kflex to create bootstrap secret and the helm chart to create the CP would be to add an option to the adopt subcommand --create-bootstrap-secret-only - would that help?

Assuming that one has a kubeconfig for the external cluster, I guess one can use kubectl to do it... that's way I was asking for clarifications on what the secret contains

kubectl create secret generic <cluster1-bootstrap-secret> --from-file=<cluster1-key>=<cluster1-kubeconfig-path> --namespace <cluster1-bootstrap-namespace>

Since it is assumed that the bootstrap kubeconfig has only one context, you probably need first to extract a new kubeconfig only for the context of the adopted cluster, so you'd need to run first a command such as:

kubectl --kubeconfig=$SOURCE_KUBECONFIG config view --minify --flatten \ --context=$DESIRED_CONTEXT > $OUTPUT_KUBECONFIG

and then you can run the kubectl create secret generic ... command

Since a kubeconfig has a "current context", the natural default for --adopted-context is the current context of the adopted kubeconfig.

francostellari · 2024-12-17T01:25:48Z

chart/crds/crds.yaml

+                  BootstrapSecretRef contains a reference to the kubeconfig used to bootstrap adoption of
+                  an external cluster
+                properties:
+                  inClusterKey:


Can you please clarify what is inClusterKey?

Perhaps I missed, but I did not see a documentation of what the bootstrap secret should look like. This is useful if somebody wants/needs to create it without using kflex.

You may see the defaults being used for the bootstrap secret in cmd/kflex/adopt/adopt.go. The name is defined in:

func GenerateBoostrapSecretName(cpName string) string { return fmt.Sprintf("%s-bootstrap", cpName) }

namespace is SystemNamespace = "kubeflex-system" and key is KubeconfigSecretKeyInCluster = "kubeconfig-incluster"

To further clarify, inClusterKey is the name of the secret generated by kflex from the bootstrap secret... right?

Since you have a default, can you make it not required?

To further clarify, inClusterKey is the name of the secret generated by kflex from the bootstrap secret

No, inClusterKey is the name of the key used for the kubeconfig, which is kubeconfig-incluster. The name of the secret would be <cp-name>-bootstrap. You do not have to provide these values in the ControlPlane CR, they are optional.

(A) Since the golang source re-uses the existing type SecretReference, regularity is implied here. It is a reference to the same sort of thing. Same schema for contents. (And, BTW, the expectation(s) on Secret contents should be documented in the comment on the SecretReference type.) I am not sure yet whether this usage really is regular.

(B) @pdettori, what do you mean by "You do not have to provide these values in the ControlPlane CR, they are optional"? Which values are optional?

MikeSpreitzer · 2024-12-20T20:43:44Z

api/v1alpha1/controlplane_types.go

+	BootstrapSecretRef *SecretReference `json:"bootstrapSecretRef,omitempty"`
+	// expiration time for token of adopted cluster
+	// +optional
+	AdoptedTokenExpirationSeconds *int64            `json:"adoptedTokenExpirationSeconds,omitempty"`


(A) If the type is "external", shouldn't this field's name start with "External" rather than "Adopted"?

(B) Is this parameter really only relevant to the case of external type? Isn't this a question that arises for every type (possibly excepting host)?

MikeSpreitzer · 2024-12-20T20:54:57Z

chart/templates/operator.yaml

@@ -591,7 +591,7 @@ spec:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
-        - --v={{.Values.verbosity}}
+        - --v=0


Is this better than simply omitting the -v ?

MikeSpreitzer · 2024-12-20T21:00:48Z

docs/users.md

@@ -280,6 +280,8 @@ clusters registration and support for the [`ManifestWork` API](https://open-clus
 - vcluster: this is based on the [vcluster project](https://www.vcluster.com) and provides the ability to create pods in the hosting namespace of the hosting cluster.
 - host: this control plane type exposes the underlying hosting cluster with the same control plane abstraction
 used by the other control plane types.
+- external: this control plane type exposes an external cluster with the same control plane abstraction


I suspect that the intended significance of "external" may not be clear to readers who do not already know what we are trying to say. I would have words here explaining that this type of ControlPlane is a representation of a preexisting cluster, one that was not created by KubeFlex. And not the hosting cluster.

MikeSpreitzer · 2024-12-20T21:17:10Z

docs/users.md

+In this section, we will show an example of creating an external control plane to integrate
+a kind cluster named `ext1` into the same Docker network as your Kubeflex hosting cluster. This setup 
+requires that both clusters are accessible from each other within the shared Docker network.


I find this confusing. It says that the purpose is "to integrate a kind cluster named ext1 into the same Docker network as your Kubeflex hosting cluster" BUT it also says that they already have to be on the same Docker network. So the integration is already done?

I expected that the main issue here is the creation of a ControlPlane that represents a preexisting cluster.

MikeSpreitzer · 2024-12-20T22:06:02Z

docs/users.md

+docker network inspect kind | jq '.[].Containers | to_entries[] | .value.Name'
+```
+
+The output will list the container names for your clusters. For example:


"For example:" makes me expect that what's next is an example of the output from the above command. But that is not actually what comes next.

MikeSpreitzer · 2024-12-20T22:10:58Z

docs/users.md

+docker network ls
+```
+
+Inspect any specific network to confirm its configuration using docker network inspect <network_name>.


Isn't there a simpler way? Looking at the output of docker inspect cluster1-control-plane, for example, I see the following in the NetworkSettings.Networks.kind section (and "kind" is the only entry in NetworkSettings.Networks):

"DNSNames": [ "cluster1-control-plane", "2e9a7ee1c5fe" ]

MikeSpreitzer · 2024-12-20T22:13:02Z

docs/users.md

+
+In this section, we will show an example of creating an external control plane to integrate
+a kind cluster named `ext1` into the same Docker network as your Kubeflex hosting cluster. This setup 
+requires that both clusters are accessible from each other within the shared Docker network.


Isn't it true that Docker supports multiple networks, so "the" shared network is not a general statement? Isn't it true that the real constraint here is that the "external" cluster and the KubeFlex hosting cluster are both on the same Docker network, regardless of which one that is?

MikeSpreitzer · 2024-12-20T22:15:17Z

docs/users.md

+To set up the external cluster ext1 as a control plane named cpe, use the following command:
+
+```shell
+$ kflex adopt -c kind-ext1 -u https://ext1-control-plane:6443


The meanings of "c" in -c and "u" in -u are not really obvious and I am never going to remember them. Since this is establishing a pattern for readers to follow, I would use the long flag names here.

MikeSpreitzer · 2024-12-20T22:17:49Z

docs/users.md

+
+### External clusters on a different host
+
+If the external cluster operates on a different host and your kubeconfig context 


Is the "operates on a different host" really a critical precondition for not needing to specify a URL override?

MikeSpreitzer · 2024-12-20T22:20:07Z

docs/users.md

@@ -314,6 +316,12 @@ To create a control plane of type `host` run the command:
 kflex create cp4 --type host
 ```



Users of the API are going to need an adequate explanation too. The documentation below for CLI users omits all mention of the bootstrap secret.

MikeSpreitzer · 2024-12-20T22:21:18Z

docs/users.md

@@ -314,6 +316,12 @@ To create a control plane of type `host` run the command:
 kflex create cp4 --type host
 ```

+To create a control plane of type `external` with the required options, run the command: 


This command will also create a Secret object that lingers forever, right? If so then that seems like something worth mentioning. And if the Secret is eventually automatically deleted, users should be told about its temporary existence.

MikeSpreitzer · 2024-12-23T17:37:11Z