Unauthorized error when refreshing state

[lbrictson]

The argocd UI shows "Unauthorized" as the error when refreshing deployment state of remote kubernetes clusters, this is happening when querying remote EKS clusters using IAM auth.

I believe the issue is the auth token that is being cached is expiring before it is refreshed. This is consistently repeatably and happens on a deployment about 5 times per hour.

Steps to replicate

1. Install argocd in cluster A
2. Give argocd access to deploy to Cluster B (Cluster B is an EKS cluster in AWS using IAM auth)
3. Do a deployment, refresh the page every few minutes and you will see the unauthorized error

I think we need to be more aggressive about refreshing the IAM auth token for EKS clusters.

[alexmt]

Argo CD does not cache token. If AWS IAM authentication is configured Argo CD generates appropriate K8S config and relies on kubectl/k8s client:

argo-cd/pkg/apis/application/v1alpha1/types.go

Lines 763 to 771 in 1deeada

	return &rest.Config{
	Host: c.Server,
	TLSClientConfig: tlsClientConfig,
	ExecProvider: &api.ExecConfig{
	APIVersion: "client.authentication.k8s.io/v1alpha1",
	Command: "aws-iam-authenticator",
	Args: args,
	},
	}

I suspect issue might be in aws-iam-authenticator: kubernetes-sigs/aws-iam-authenticator#30

@lbrictson is would help if you could attach relevant application-controller logs .

[jessesuen]

I suspect it's this: kubernetes-sigs/aws-iam-authenticator#133. I can't tell which version this was fixed in, but the version of aws-iam-authenticator that we use, predates October 13th, when this was merged.

[lbrictson]

so maybe simply updating the aws-iam-authenticator binary will yield a fix?

[jessesuen]

Yes, I believe so. I took a look at the fix. The version of aws-iam-authenticator we are bundling does not set expiration of tokens so go-client does not seem to know to refresh it until it’s too late. Their fix is to set a refresh time at an interval less than the one AWS uses (14 minute refresh vs. a 15 minute token).

[jessesuen]

@lbrictson if you replace aws-iam-authenticator with the v0.4.0-alpha.1 release, will things start working? You would have to kubectl cp the binary to the argocd-server and application-controller pods and replace the binary.

https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/tag/0.4.0-alpha.1

If things work, I think we can ship Argo CD with v0.4.0-alpha.1 aws-iam-authenticator release, so long as nothing else is regressing. I don't have an EKS cluster readily available to test with at the moment.

[lbrictson]

@jessesuen that seems doable, i'll drop that into the containers and see what happens. The issue happens several times per hour so it should be easy to see the effects of.

I'll report back later today with the results

[jessesuen]

Fixed in #948