Eks 1.22-1.21 Unseal IRSA support (Volume Service Token)

[fbuinosquy1985]

Eks 1.21/ 1.22 requires that we have to mount the Service Account Token using a Volume and VolumeMount

Example:
#extraVolumeMounts:
# - name: aws-iam-token
# readOnly: true
# mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
#extraVolumes:
# - name: aws-iam-token
# projected:
# defaultMode: 420
# sources:
# - serviceAccountToken:
# audience: sts.amazonaws.com
# expirationSeconds: 86400
# path: token

Currently i have set on my vault service account the following annotation

eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXXXX:role/vault-unseal

to unseal Vault obtaining the SSM key from AWS.

Is there a way to mount the Volume and the VolumeMount of that token currently on the Helm Chart

[tvoran]

Hi @fbuinosquy1985, if you're using IRSA, then EKS should mount the token for you in the vault pods if their service account is annotated with an IRSA role as you mentioned. If the AWS side is setup correctly, then the pod spec will be mutated by amazon to include environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, which will be picked up by the awskms logic in vault.

More details: #368 (comment)

[fbuinosquy1985]

@tvoran it was working with 1.19 and 1.20 , but with 1.21 and 1.22 it doesnt work because by default Eks 1.21/1.22 doesnt mount the IRSA token automatically after it expires , https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

[tvoran]

Ah, then it sounds like this section of the docs may be of interest: https://www.vaultproject.io/docs/auth/kubernetes#use-local-service-account-token-as-the-reviewer-jwt

Vault versions prior to 1.9.3 would only read the local service account token once, so when that token expires (as it does now with k8s 1.21+) authentication would fail. Versions 1.9.3 and later re-read the token as needed.

[fbuinosquy1985]

@tvoran No sorry for not being clear the issue is with the aws kms key vault unseal process that cannot be obtained if the account doesn't have a way to obtain their aws role

[tvoran]

Hmm, I believe the IRSA token has been mounted as a projected volume since at least EKS 1.13, so I don't know of any changes in 1.21 that would affect it. I tried the configuration I described in #368 (comment) with EKS 1.22 and auto-unseal still works as expected. And the IRSA token is refreshed in place once it reaches ~80% of its expiration.

If you don't want the EKS pod identity webhook to mount the projected volume IRSA token for you, I'd suggest trying the server.volumes and server.volumeMounts options in the chart. They allow you to define volumes and volumeMounts in the same format and flexibility as you would in a Pod spec. (And also use the server.extraEnvironmentVars option to set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE.)

It would be helpful if you provided the vault config, helm chart config, errors in the logs, etc. to debug this further.