How to prevent Wait(n=1) would exceed context deadline in a kustomize-controller reconcilliation?

[FelipeEmerimRandon]

For some reason, kustomize-controller will sometimes tell that some resources have an 'Unknown' status and the status message shows: health check failed after 3m0.020770912s: timeout waiting for: [Resource/some-resource status: 'Unknown': client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

I would like to know if there is anything I can tweak on Flux to prevent this, or if this is the kubeAPI rate-limiting our kustomize-controller.

We usually see this error when a kustomization times out after 3min which can happen because a deployment has not finished its rollout. The odd thing is that flux reports the status mentioned above in other resources like, for example, serviceAccounts, virtualServices.

We have about 200 resources managed by this flux shard, and are using the default settings of kustomize-controller. The resources are split across multiple kustomizations.

We are using AKS 1.29.4 and flux v2.3.0

[stefanprodan]

What’s the .spec.timeout value in your Flux Kustomization?

[FelipeEmerimRandon]

These are our kustomizations intervals and timeout:

interval: 10m0s # detect drift and undo kubectl edits every hour
wait: true # wait for all applied resources to become ready
timeout: 3m0s # give up waiting after three minutes
retryInterval: 2m0s # retry every two minutes on apply or waiting failures

We expect it to timeout since the rollout does take longer than the timeout, but normally it times out correctly with the message that the deployment has the status inProgress not with status unknown and the client rate limiter message.

[joaocc]

Hi. We also started receiving these messages.
The only semi-reliable workaround is to kill flux pods and wait for them to check things again.
Also using EKS 1.29 and flux v2.3.0

[stefanprodan]

Set .spec.retryInterval instead of killing the pod.