Pod is using stale tokens

[albertschwarzkopf]

Hi,

the "Bound Service Account Token Volume" is graduated to stable and enabled by default in Kubernetes version 1.22.
I am using "aws-for-fluent-bit:2.23.4" in AWS EKS 1.22 and I have checked, if it is using stale tokens (regarding https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html and https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#troubleshooting-boundservicetoken).

So when the API server receives requests with tokens that are older than one hour, then it annotates the pod with "annotations.authentication.k8s.io/stale-token". In my case I can see the following annotation. E.g.:

annotations.authentication.k8s.io/stale-token subject: system:serviceaccount:kube-system:fluentbit-oidc, seconds after warning threshold: 969

Fluent Bit Version Info

aws-for-fluent-bit:2.23.4

Cluster Details

AWS EKS 1.22
aws-for-fluent-bit:2.23.4 deployed as Daemonset

Steps to reproduce issue

• Enable EKS Audit Logs
• Query CW Insights (select cluster log group):

fields @timestamp
| filter @message like /seconds after warning threshold/
| parse @message "subject: *, seconds after warning threshold:*\"" as subject, elapsedtime

[matthewfala]

Hi @albertschwarzkopf,
We're working on getting a solution for stale-token issue accepted upstream. Please see: fluent/fluent-bit#4487

Hopefully this issue will be resolved in the next or following aws-for-fluent-bit release. Currently the fix is awaiting further review and merge.

[PettitWesley]

We released a cherry-pick of the fix in our latest release, please try it: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.25.0