CAPI AWS Cluster - Sveltos Addons fail due to wrong CAPI kubeconfig

[dazmc]

Trying to deploy various addons to managed EKS cluster created via CAPI results in in consistent results

sveltos version - 0.36.0
cluster-api version - v1.8.1
infrastructurei-aws version - v2.6.1

SLACK DISCUSSION/TROUBLESHOOTING

1. Deploy EKS cluster and then configure for capi and sveltos as mgmt cluster

# capi
clusterawsadm bootstrap iam create-cloudformation-stack --region us-east-1
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
export AWS_REGION=us-east-1
export EKS=true
export EXP_MACHINE_POOL=true
export CAPA_EKS_IAM=true
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large
export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=capi-eks
aws ec2 create-key-pair --key-name capi-eks --region us-east-1 --query 'KeyMaterial' --output text > capi-eks.pem
clusterctl init --infrastructure aws

# sveltos 
# Note: i use chartproxy to proxy https to oci 
helm upgrade --install projectsveltos oci://chartproxy.container-registry.com/projectsveltos.github.io/helm-charts/projectsveltos -n
projectsveltos --create-namespace

2. deploy capi workload cluster

kubectl apply -f capi-eks.yaml

3. label the cluster and deploy simple clusterprofile to the managed cluster

kubectl label cluster capi-eks env=prod
kubectl apply -f clusterprofile_podinfo

# contents
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: podinfo
spec:
  clusterSelector:
    matchLabels:
      env: prod
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://stefanprodan.github.io/podinfo/
    repositoryName:   podinfo
    chartName:        podinfo/podinfo
    chartVersion:     6.7.0
    releaseName:      podinfo-latest
    releaseNamespace: podinfo
    helmChartAction:  Install

4. Nothing happens and errors appear in the logs

# addon controller

E0829 14:21:56.708112       1 clusterproxy.go:65] "BuildConfigFromFlags" err="error loading config file \"/tmp/kubeconfig1256201832\": couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }" cluster="default/capi-eks"
I0829 14:21:56.708293       1 resourcesummary_collection.go:55] "failed to collect ResourceSummaries from cluster: default/capi-eks BuildConfigFromFlags: error loading config file \"/tmp/kubeconfig1256201832\": couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }"
E0829 14:22:06.721660       1 clusterproxy.go:65] "BuildConfigFromFlags" err="open /tmp/token-file: no such file or directory" cluster="default/capi-eks"
I0829 14:22:06.721839       1 resourcesummary_collection.go:55] "failed to collect ResourceSummaries from cluster: default/capi-eks BuildConfigFromFlags: open /tmp/token-file: no such file or directory"
E0829 14:22:16.738122       1 clusterproxy.go:65] "BuildConfigFromFlags" err="open /tmp/token-file: no such file or directory" cluster="default/capi-eks"
I0829 14:22:16.738204       1 resourcesummary_collection.go:55] "failed to collect ResourceSummaries from cluster: default/capi-eks BuildConfigFromFlags: open /tmp/token-file: no such file or directory"
E0829 14:22:26.752496       1 clusterproxy.go:65] "BuildConfigFromFlags" err="open /tmp/token-file: no such file or directory" cluster="default/capi-eks"
I0829 14:22:26.752795       1 resourcesummary_collection.go:55] "failed to collect ResourceSummaries from cluster: default/capi-eks BuildConfigFromFlags: open /tmp/token-file: no such file or directory"
I0829 14:22:36.110929       1 main.go:562] "memory stats" logger="memory-usage" Alloc (MiB)=31 TotalAlloc (MiB)=142 Sys (MiB)=63 NumGC=19
I0829 14:22:42.079536       1 reflector.go:808] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:232: Watch close - *v1beta1.Machine total

# show addons is blank

sveltosctl show addons
+------------------+---------------+-----------+----------------+---------+-------------------------------+------------------------+
|     CLUSTER      | RESOURCE TYPE | NAMESPACE |      NAME      | VERSION |             TIME              |        PROFILES        |
+------------------+---------------+-----------+----------------+---------+-------------------------------+------------------------+
+------------------+---------------+-----------+----------------+---------+-------------------------------+------------------------+


# managed cluster status show failed

Status:
  Dependencies:  no dependencies
  Feature Summaries:
    Failure Message:    Kubernetes cluster unreachable: open /tmp/token-file: no such file or directory
    Feature ID:         Helm
    Hash:               l5DI7wv7fHhZjngMV9n9zKGtfATb2A35ABJ/9uej0bA=
    Last Applied Time:  2024-08-29T15:03:25Z
    Status:             Failed
  Helm Release Summaries:
    Release Name:       podinfo-latest
    Release Namespace:  podinfo
    Status:             Managing
Events:                 <none>

5. Capi kubeconfigs

There are 2 kubeconfigs created for a workload cluster, it would appear it is the first one that sveltos uses and this secret has multiple entries kv which is causing the issue. As only one kv in the secret has the kubeconfig, i guess sveltos is using the wrong kv in the secret. This causes the errors as seen in the logs and the inconsistent behavior.

#  capi-eks-kubeconfig
{
  "relative": "YXBpVmVyc2l....",
  "token-file": "azhzLWF3c....",
  "value": "YXBpVmVyc2K....."
}

and

# capi-eks-user-kubeconfig

{
  "value": "YXBpVmVyc2lvbj....K"
}

6. workaround is to register the cluster-api cluster as a sveltos cluster.

This results in a secret with a single kv with the kubeconfig to connect to the workload cluster - which sveltos uses for connectivity

sveltosctl register cluster --namespace=default --cluster=kingston --fleet-cluster-context=arn:aws:eks:us-east-1:XXXXXXXXXX:cluster/default_capi-eks-control-plane --labels=env=prod,region=us