403 unauthorized - BoundServiceAccountToken and token refresh support.

[cilindrox]

Kubernetes version 1.21 defaults to enabling the beta version of BoundServiceAccountTokenVolume by default. Service account tokens now have an expiration of one hour, which means that clients that rely on these tokens must refresh the tokens within an hour.

We've run into keel getting 403 API errors after the tweaked 90d refresh interval has expired on EKS. This was fixed by a Pod restart, thus extending the period to another 90d, but this is a hack that will become bothersome for those environments that use the default 1h period.

It looks like as if the Kubernetes client SDK for Go automatically refresh tokens within the required time frame, so supporting newer k8s versions might be a matter of upgrading the SDK.

Sample trace of the error once the refresh interval is up:

1 reflector.go:126] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:94: Failed to list *v1.DaemonSet: Unauthorized

[georgecodes]

Looking at 1.17.0-rc1 release:

keel/go.mod

Line 6 in 36bbafc

k8s.io/api => k8s.io/api v0.16.10

suggests that the k8s api has been upgraded to v0.16.10

According to this

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.21

The client needs to be at least v0.15.7 to refresh tokens, so upgrading your keel version to 1.17.0-rc1 should fix this.

[plumdog]

In case it helps anyone, the Keel image tag is 0.17.0-rc1 (that is, 0. not 1.). So, eg if using helm, you need to set:

image:
  tag: 0.17.0-rc1

Can review all available images here: https://hub.docker.com/r/keelhq/keel/tags