Declarative creation of clusters

[exocode]

Summary

Is there a way how to

1. create a cluster via `kubectl apply -f cluster.yml``
2. "connect" or "point" a kubec-config to the ArgoCD created cluster? Something like kubeConfigSecretKeyRef: "cluster-details-my-cluster-kube-config"?

Motivation

I have the ArgoCD server running and wanna define a Cluster without the CLI. I wanna practice GitOps, so I have to declare and commit my ArgoCD-cluster config in Git.

In the CLI I could do: argocd cluster add but how to do that with a Kubernetes manifest?

I didn't found how to create that Cluster declarative. I found how to create Repositories, and Projects. But nothing for something like kind: cluster. I have to manually intervent my cluster which breaks the GitOps practice.

I am creating my clusters with Crossplane. Crossplane saves the kubeconfig of it's created cluster in a Secret which looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-details-my-cluster
  namespace: default
  uid: 50c7acab-3214-437c-9527-e66f1d563409
  resourceVersion: '12868'
  creationTimestamp: '2022-01-06T19:03:09Z'
  managedFields:
    - manager: crossplane-civo-provider
      operation: Update
      apiVersion: v1
      time: '2022-01-06T19:03:09Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:kubeconfig: {}
        f:type: {}
  selfLink: /api/v1/namespaces/default/secrets/cluster-details-my-cluster
data:
  kubeconfig: >-
    YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VKbFJFTkRRVkl5WjBGM1NVSkJaMGxDUVVSQlMwSm5aM0ZvYTJwUFVGRlJSRUZxUVdwTlUwVjNTSGRaUkZaUlVVUkVRbWh5VFROTmRHTXlWbmtLWkcxV2VVeFhUbWhSUkVVeVRrUkZNRTlVVlROT1ZFbDNTR2hqVGsxcVNYZE5WRUV5VFZScmQwMXFUWGxYYUdOT1RYcEpkMDFVUVRCTlZHdDNUV3BOZVFwWGFrRnFUVk5_SHORTENED
type: Opaque

The data.kubeconfig content is a regular bas64 encoded kubeconfig, so it's easy to decode, like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHd_SHORTENED
    server: https://MY.IP.TO.K8S:6443
  name: my-cluster
contexts:
- context:
    cluster: my-cluster
    user: my-cluster
  name: my-cluster
current-context: my-cluster
kind: Config
preferences: {}
users:
- name: my-cluster
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJQS9adEZFT1Avcnd3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOalF4TkRrMU56VXlNQjRYRFRJeU1ERXdOakU1TURJek1sb1hEVEl6TURFdwpOakU1TURJek1sb3dNREVYT_SHORTENED
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpJNlVhTDlLem9yL1VpdzlXK1NNUTAxV1BES2ZIK_SHORTENED

Proposal

If that feature does not exist I propose a CRD for kind: cluster which can be applied by kubectl apply -f cluster.yml

How do you think this should be implemented?

Something like this:

kind: cluster
metadata: 
  name: my-cluster-config
  namespace: argocd 
  server: https://74.220.24.192:6443 # target cluster
  credentialsType: kubeconfig # maybe others need .crt or else
config:
  valueFrom:
    secretKeyRef:
      name: my-cluster-kubeconfig-secret # secret created by crossplane
      namespace: default # namespace of secret
      key: kubeconfig # 
  

[chetan-rns]

Hi @exocode, Argo CD uses secrets to represent clusters. You can find more details here https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters

[exocode]

@chetan-rns Thank you, I have to investigate that deeper. Nevertheless it would be really nice to have a more "simpler setup" (especially for Crossplane).

Because - if I understand that correct, and please bear in mind, that I am a Argo noob - we have to "remap" a regular kubeconfig somehow to match the ArgoCD cluster definitions, which are scattered in different secrets.

That opens another question to me as well another feature request if not already existing:

When my cluster is created and ArgoCD has configred (somehow Gitops like) for accessing the cluster, I then do have to change ALL my Application-manifest IPs ( spec.destinations.server ), to point to the new cluster?

Or is there an Argo-way to centrally change the server IPs (for example in a ConfigMaps) and reference them in Application like so:

Something like this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
spec:
  description: production deployment
  serverDestinationConfigRefs: # <<< FEATURE REQUEST?
    - name: "production-cluster-config-map" # <<< points to configMap containing cluster IPs

That last lines avioding redundancy in configuration.
Or how others handle such cases? Are they changing their IPs manually and recommit them? That would also break (in my opinion) GitOps.

Please help me to dissolve that knot in my brain. Thank you very much in advance

[rouke-broersma]

@exocode you can use the cluster name instead of the IP/url instead.

[janwillies]

Something like this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
spec:
  description: production deployment
  serverDestinationConfigRefs: # <<< FEATURE REQUEST?
    - name: "production-cluster-config-map" # <<< points to configMap containing cluster IPs

note that this kind of referencing is possible with crossplanes provider-argocd: https://doc.crds.dev/github.com/crossplane-contrib/provider-argocd/projects.argocd.crossplane.io/Project/[email protected]#spec-forProvider-destinations

[morningspace]

I have similar requirement. It looks right now I can only use argocd CLI to add cluster. Is there way to make it using kubectl apply? It looks mostly it's encapsulated inside a secret, but that could be hacky to manually create this secret?

[rouke-broersma]

I have similar requirement. It looks right now I can only use argocd CLI to add cluster. Is there way to make it using kubectl apply? It looks mostly it's encapsulated inside a secret, but that could be hacky to manually create this secret?

This is not true. "Declarative Setup - Argo CD - Declarative GitOps CD for Kubernetes" https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters

[exocode]

I modified a script of @janwillies so, that my kubeconfig-Secret - which was created by my Crossplane cloud-provider (provider-civo) - gets mapped to match the requirements of argoCD secrets. It also assigns the kubeconfig values in the data-root of that secret, so it can be consumed by other resources.

The result looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: kubeconfig-demo-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
  annotations:
    managed-by: argocd.argoproj.io
data:
  bearerToken: 'ARGOCD_USER_TOKEN_wljsdf7s&5$56' # optional
  caData: >-
    TFMwdExTMUNSVWR......TVU5CVkVVdExTMHRMUW89
  certData: >-
    TFMwdExTMUNSV.......TMHRDZz09
  config: >-
    eyJiZWFyZXJUb2tlbiI6IiI.......89In19
  keyData: >-
    TFMwdExTMUNSVWRKVGlCRlF5QlFVa2xX......FvPQ==
  name: ZGVtby1jbHVzdGVy
  server: aHR0cHM6Ly83NC4yMjAuMjcuMjQ3OjY0NDM=
  serverName: ZGVtby1jbHVzdGVy
type: Opaque

https://github.com/exocode/crossargo-sync

Simply change the namespace which should be observed for kubeconfig secrets at the bottom of deployment.yaml. Then apply it.
How it works:

if a data.kubeconfig value is found in any of the secrets in the "observed" namespace. The ArgoCD secret will be created.

After that, your cluster is automatically listed in ArgoCD and can be consumed by AppProjects and Applications

What it does not:

• It is not updating or deleting secrets

I know that is a bit clunky, because I am not a Go coder, but it does exaclt what I needed. Maybe some of the ArgoCD maintainers see what we try to achieve and what we need.

[morningspace]

@exocode I am reading your code, and wonder where the bearerToken comes from.

When I inspect the secrets created in my env, it looks something as below:

apiVersion: v1
kind: Secret
metadata:
  name: foo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
  annotations:
    managed-by: argocd.argoproj.io
data:
  config: eyJiZW...
  name: bmVlZG...
  server: aHR0cH...
type: Opaque

And the config can be decoded as below:

{
  "bearerToken": "eyJhbG",
  "tlsClientConfig": {
    "insecure": true
  }
}

I don't know how to prepare the bearerToken, so that I can create the secret by myself.

[exocode]

That token comes from an ArgoCD user created to access and create ArgoCD ressources for this Crossplane extension: https://github.com/crossplane-contrib/provider-argocd
The steps to produce that user are here:
https://github.com/crossplane-contrib/provider-argocd/blob/main/README.md

TLDR:

ARGOCD_ADMIN_SECRET=$(kubectl view-secret argocd-initial-admin-secret -n argocd -q)
echo $ARGOCD_ADMIN_SECRET
ARGOCD_ADMIN_TOKEN=$(curl -X POST -k -H "Content-Type: application/json" --data '{"username":"admin","password":"'$ARGOCD_ADMIN_SECRET'"}' https://localhost:8443/api/v1/session | jq -r .token)
echo $ARGOCD_ADMIN_TOKEN

kubectl patch configmap/argocd-cm -n argocd --type merge -p '{"data":{"accounts.provider-argocd":"apiKey, login"}}'
kubectl patch configmap/argocd-rbac-cm -n argocd --type merge -p '{"data":{"policy.default":"role:admin"}}'

ARGOCD_PROVIDER_USER="provider-argocd"
ARGOCD_TOKEN=$(curl -s -X POST -k -H "Authorization: Bearer $ARGOCD_ADMIN_TOKEN" -H "Content-Type: application/json" https://localhost:8443/api/v1/account/$ARGOCD_PROVIDER_USER/token | jq -r .token)
echo $ARGOCD_TOKEN
kubectl create secret generic argocd-credentials -n crossplane-system --from-literal=authToken="$ARGOCD_TOKEN"

[sharadhirao]

Hi @exocode ,
Could you let me know if this issue has been solved? I also use a similar kubeconfig file you mentioned in the first message. For now I am using argocd cluster add.

[exocode]

Hi @sharadhirao,

no, I still use a "last manual step", but my latest message shows exactly what I'm doing. It seems not that popular.
For someone who is a Go programmer this must be a trivial task, but I am not one.
Everyone's talking GitOps and automation, but that part of automation seems not so interesting. Interesting isn't it ? :-)

[sharadhirao]

Hi @exocode,
Thanks a lot for reply. Yes true. I am using GitHub to set all things needed for ArgoCD. I wanted to store the cluster secret with kubeconfig file so that argocd syncs it to have GitOps strategy for secret also. Let me check if I can automate this cluster addition somehow.

[pablo-de]

I subscribe to the @exocode request, I am working on an automation and it would be very useful to have this crd.

[joebowbeer]

Close this now that Crossplane's provider-argocd can register the cluster?

Demo: https://github.com/adavarski/k3d-crossplane-vcluster-playground

This repo contains a Crossplane composition to spin up virtual Kubernetes clusters which are automatically registered at ArgoCD (via provider-argocd)

[rouke-broersma]

Close this now that Crossplane's provider-argocd can register the cluster?

Demo: https://github.com/adavarski/k3d-crossplane-vcluster-playground

This repo contains a Crossplane composition to spin up virtual Kubernetes clusters which are automatically registered at ArgoCD (via provider-argocd)

The ask in the issue is imo more that the argocd cluster secret supports a kubeconfig file so transformation to argocd format is not necessary. Crossplane is external tooling,which we do not use.

[joebowbeer]

@rouke-broersma the OP is using Crossplane and I wonder if their issue is now solved via provider-argocd.

If support for kubeconfig is what is required, I think it maybe helpful to update the title and description. Or close this as a dup of #4651 ?

There are a handful of issues related to cluster secrets. Most of them are old and it's difficult for me to determine what is being asked for and what remains to be done.

[rouke-broersma]

Sorry you're right, I misunderstood!

[jonashackt]

If anybody needs a full guide on how to argocd-provider to register Crossplane created K8s clusters in Argo, I created a thorough answer here. It all boils down to use the argocd-providers Cluster object like this:

apiVersion: cluster.argocd.crossplane.io/v1alpha1
kind: Cluster
metadata:
  name: argo-reference-deploy-target-eks
  labels:
    purpose: dev
spec:
  forProvider:
    config:
      kubeconfigSecretRef:
        key: kubeconfig
        name: eks-cluster-kubeconfig # Secret containing the kubeconfig to access the Crossplane created EKS cluster
        namespace: default
    name: deploy-target-eks # name of the Cluster registered in ArgoCD
  providerConfigRef:
    name: argocd-provider

Here's also a full example project: https://github.com/jonashackt/crossplane-argocd