[Bug] Report sidecar failed to run on k8s

[tthvo]

Current Behavior

When installing the chart with some report sidecar replicas, the sidecar pods failed to start with the container throwing permission denied error.

When getting pod status:

$ kubectl get pods
NAME                                            READY   STATUS             RESTARTS     AGE
release1-cryostat-v4-59565cf4fd-hwlbb           4/4     Running            0            2m25s
release1-cryostat-v4-db-75dd7f5c4c-l4wd5        1/1     Running            0            2m25s
release1-cryostat-v4-reports-69dfd7cf94-sf6n7   1/2     CrashLoopBackOff   4 (7s ago)   2m25s
release1-cryostat-v4-storage-7494ccb454-rhv8b   1/1     Running            0            2m25s

When extracting the oauth2-proxy container log:

$ kubectl logs -f pods/release1-cryostat-v4-reports-69dfd7cf94-sf6n7 -c cryostat-reports-authproxy
[2024/12/02 22:44:28] [main.go:59] ERROR: Failed to initialise OAuth2 Proxy: could not validate htpasswd: could not load htpasswd file: could not open htpasswd file: open /etc/oauth2_proxy/basicauth/htpasswd: permission denied
[2024/12/02 22:44:28] [oauthproxy.go:128] using htpasswd file: /etc/oauth2_proxy/basicauth/htpasswd

Expected Behavior

The report sidecar pods start successfully.

Steps To Reproduce

1. Have an existing k8s cluster, for example, minikube (i.e. minikube start --cni calico --cpus=8 --driver=kvm2)

2. With latest main commit, install the chart:

   helm install release1 charts/cryostat/ --set reports.replicas=1

3. Observe other components are healthy except the report pod.

Environment

- OS: Fedora 40
- Environment: Kubernetes v1.30.0
- Version: Latest chart snapshot on `main`

Anything else?

I haven't yet tested on OpenShift but it seems to fails on kubenetes with oauth2-proxy. I tried kinD and minikube.

[tthvo]

I does seem like configurations for basic auth for Cryostat itself also failed (i.e. on Kubernetes):

$ helm install cryostat ./charts/cryostat/ --set authentication.basicAuth.enabled=true,authentication.basicAuth.secretName=htpasswd,authentication.basicAuth.filename=htpasswd --set reports.replicas=1

# Cryostat pod
$ kubectl logs -f pods/cryostat-v4-6c8b8c5646-2vqr5
Defaulted container "cryostat-authproxy" out of: cryostat-authproxy, cryostat, cryostat-grafana, cryostat-jfr-datasource
[2024/12/02 23:37:23] [main.go:72] WARNING: You are using alpha configuration. The structure in this configuration file may change without notice. You MUST remove conflicting options from your existing configuration.
[2024/12/02 23:37:23] [oauthproxy.go:128] using htpasswd file: /etc/oauth2_proxy/basicauth/htpasswd
[2024/12/02 23:37:23] [main.go:59] ERROR: Failed to initialise OAuth2 Proxy: could not validate htpasswd: could not load htpasswd file: could not open htpasswd file: open /etc/oauth2_proxy/basicauth/htpasswd: permission denied

# Report pod
$ kubectl logs -f pods/cryostat-v4-reports-5b8894f4cc-xjjmk 
Defaulted container "cryostat-reports-authproxy" out of: cryostat-reports-authproxy, cryostat-reports
[2024/12/02 23:38:38] [oauthproxy.go:128] using htpasswd file: /etc/oauth2_proxy/basicauth/htpasswd
[2024/12/02 23:38:38] [main.go:59] ERROR: Failed to initialise OAuth2 Proxy: could not validate htpasswd: could not load htpasswd file: could not open htpasswd file: open /etc/oauth2_proxy/basicauth/htpasswd: permission denied

[tthvo]

Observations

After a bit of investigations, I think the root cause is that the oauth2-proxy container is running as both non-root user & group, while the mounted volume is owned by root user & group. Additionally, the permission is 0440, which causes the container (i.e. treated as "others") to be denied access.

cryostat-helm/charts/cryostat/templates/cryostat_deployment.yaml

Line 211 in b91e9bd

defaultMode: 0440

Suggestions

There are couple ways I tried that worked:

• Set the permission mode to 0444, which allows "others" to read. This seems to be simpler way.
• Set fsGroup on the securityContext of the pod. We need to be careful because OpenShift does have restrictions on fsGroup value (i.e. reported on the namespace annotation).

On OpenShift, I believe the Security Context Constraint system does set fsGroup on pods so it works fine.

Debugging

After installing the chart, I followed these steps to get a shell on the node (i.e. with single-node minikube) for debugging:

$ kubectl debug -it node/minikube --image=busybox:1.28
Creating debugging pod node-debugger-minikube-zxh24 with container debugger on node minikube.
If you don't see a command prompt, try pressing enter.
/ # 
/ # chroot /host
bash-5.2# docker ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED          STATUS          PORTS     NAMES
...output-omitted...
786df5ca9262   quay.io/cryostat/jfr-datasource               "/opt/jboss/containe…"   4 hours ago      Up 4 hours                k8s_cryostat-jfr-datasource_release1-cryostat-v4-768b7bc5d8-jphqp_default_8933794f-a605-42c2-a495-bc9ce1e7a3cd_2
9f52b145cda9   quay.io/cryostat/cryostat-grafana-dashboard   "/run.sh"                4 hours ago      Up 4 hours                k8s_cryostat-grafana_release1-cryostat-v4-768b7bc5d8-jphqp_default_8933794f-a605-42c2-a495-bc9ce1e7a3cd_2
a66a2d168aa8   quay.io/cryostat/cryostat                     "/deployments/app/en…"   4 hours ago      Up 4 hours                k8s_cryostat_release1-cryostat-v4-768b7bc5d8-jphqp_default_8933794f-a605-42c2-a495-bc9ce1e7a3cd_2
3620502a545c   quay.io/cryostat/cryostat-reports             "/opt/jboss/containe…"   4 hours ago      Up 4 hours                k8s_cryostat-reports_release1-cryostat-v4-reports-69dfd7cf94-qgb8n_default_84ce55c0-f42e-4590-b4e3-0add50a2e81e_2
f12684b2647a   quay.io/oauth2-proxy/oauth2-proxy             "/bin/oauth2-proxy -…"   4 hours ago      Up 4 hours                k8s_cryostat-authproxy_release1-cryostat-v4-768b7bc5d8-jphqp_default_8933794f-a605-42c2-a495-bc9ce1e7a3cd_2
...output-omitted...

bash-5.2# docker top f12684b2647a -eo user,group,pid,comm
USER                GROUP               PID                 COMMAND
65532               65532               5964                oauth2-proxy