[prometheus-kube-stack] serviceaccount token is not readable for EKS 1.21 - permission denied

[rizwan-kh]

Describe the bug a clear and concise description of what the bug is.

The kube-prometheus-stack-admission-create job is failing with error - "error":"open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied"

I think this will be a generic issue with the entire chart. Amazon EKS supports IAM Roles for Service Accounts. It mounts tokens files to /var/run/secrets/eks.amazonaws.com/serviceaccount/token but to read the token we would need to set the securityContext.fsGroup field as per this document

The solution here works for individual deployment by adding the securityContext.fsGroup, however, I am not able to find a way to apply this on the entire chart and its dependency.

What's your helm version?

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

What's your kubectl version?

v1.22.0

Which chart?

prometheus-operator

What's the chart version?

33.0.0

What happened?

The kube-prometheus-stack-admission-create job is failing with the below error

W0228 13:28:10.878848       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0228 13:28:10.878925       1 client_config.go:620] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
{"error":"open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied","level":"fatal","msg":"error building kubernetes config","source":"cmd/root.go:89","time":"2022-02-28T13:28:10Z"}

The solution here works for individual deployment by adding the securityContext.fsGroup, however, I am not able to find a way to apply this on the entire chart and its dependency.

What you expected to happen?

Chart to be deployed and with no error.

How to reproduce it?

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install prom -n mon prometheus-community/kube-prometheus-stack --version 33.0.0

Enter the changed values of values.yaml?

None

Enter the command that you execute and failing/misfunctioning.

helm upgrade --install prom -n mon prometheus-community/kube-prometheus-stack --version 33.0.0

Anything else we need to know?

No response

[monotek]

you might be able to set fsgroup setting in chart version 33.1.0.
see: #1811

[rizwan-kh]

@monotek - I tried the below with same error

helm upgrade --install prom -n mon prometheus-community/kube-prometheus-stack --version 33.1.0

[rizwan-kh]

I will close this, was able to resolve and set fsGroup

# values.yaml
prometheusOperator:
  admissionWebhooks:
    patch:
      securityContext:
        fsGroup: 65534

helm upgrade --install prom -n mon prometheus-community/kube-prometheus-stack -f values.yaml --version 33.1.0