Flux compatibility - service account annotations

[chlunde]

What happened?

We sometimes see bursts of Unauthorized for calls from crossplane providers to the kubernetes apiserver. After some debugging we've found that the UID in the service account does not match the token on the file system. We noticed that the service account has a "fluxcd.io" annotation, even though it is not managed by flux. This is because crossplane copies all annotations from the ControllerConfig to the ServiceAccount object. This is probably required for ArgoCD, and it is used for IRSA in AWS.

But for flux, this does not work. We believe flux will consider this as a situation where a former Kustomization managed this object, and delete it as it is no longer managed. crossplane will quickly recreate it, but the token mounted in the container will for some time have the old UID and the API server will reject the requests. I'm not sure if this is a race condition in the kubelet when a token is rotated while a pod is created, or if it is simply not a supported way to rotate the tokens by deleting the service account while it is in use, but Kubernetes does recover from this situation after a while.

The specific annotation is "kustomize.toolkit.fluxcd.io/checksum": "3583e44b78adb421bd53fcb43edfcd5d39b287b8"

How can we reproduce it?

Flux-managed ControllerConfig, see above

Suggested fix

I think we should exclude annotations with keys fluxcd.io/ or prefix kustomize.toolkit.fluxcd.io/ when creating the deployment and serviceaccount in

crossplane/internal/controller/pkg/revision/deployment.go

Lines 174 to 177 in 1344a86

	s.Labels = cc.Labels
	s.Annotations = cc.Annotations
	d.Labels = cc.Labels
	d.Annotations = cc.Annotations

We might also want to skip key kubectl.kubernetes.io/last-applied-configuration or prefix kubectl.kubernetes.io/ as it is fairly misleading to copy it.

Long term crossplane should probably implement "bring your own service account" instead (see issue #2295 etc)

What environment did it happen in?

Crossplane version: 1.6.3, EKS 1.21, flux 0.17

[chlunde]

Workaround (thanks @haarchri): kustomize.toolkit.fluxcd.io/prune: disabled

[negz]

Cross-linking to #2468

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[wwentland]

/fresh This is still an issue that should be addressed. Hopefully we can get some related PRs merged soon.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 7 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.