Skip to content

minio ‐ operator ‐ Unable to load certs ... read‐only file system | secrets incorrectly named as operator‐console‐tls

Jump to bottom
Allan Roger Reid edited this page Nov 14, 2023 · 1 revision

1. Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir /tmp/certs/CAs: read-only file system

2. Secrets in volume tls-certificates incorrectly named as operator-console-tls

Setup operator

brew cask install multipass

multipass version

multipass find

multipass launch --name k3s-single-node --cpus 2 --mem 2048M --disk 5G 22.04

multipass list

multipass shell k3s-single-node

sudo apt update -y
sudo apt upgrade -y
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -

Show k8s running

kubectl get node -o wide

NAME              STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k3s-single-node   Ready    control-plane,master   41s   v1.27.7+k3s2   192.168.64.68   <none>        Ubuntu 22.04.3 LTS   5.15.0-87-generic   containerd://1.7.7-k3s1.27

Install krew

see https://krew.sigs.k8s.io/docs/user-guide/setup/install/ for macOS/Linux > Bash or ZSH shells

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Install kubectl minio

https://min.io/docs/minio/kubernetes/upstream/reference/kubectl-minio-plugin.html#installation

kubectl krew update
kubectl krew install minio
kubectl minio version

Deploy minio

kubectl minio init --console-tls

Set TLS

kubectl set env -n minio-operator deployment/minio-operator MINIO_CI_CD=on MINIO_CONSOLE_TLS_ENABLE=on

Get jwt

SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6IlJreHBNVWFSOGxOLXdMdGdLZU93Rm1XdWo4Si10X2lhZlVYZVdjYUlKZE0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb25zb2xlLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNzBmNjNhMTctNThiNi00YjBmLWExNTAtMmViNzM3ZTI0ZmExIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1pbmlvLW9wZXJhdG9yOmNvbnNvbGUtc2EifQ.bNHXFKH_tNbA8bxTAfFBLntVeLt2QqQ5ZSahCrUyZKsedU1-1dui_JR85U2psolwOekcG4c_REPEgzL2KxjX5Wsr0qml4h0oQ5bw6mHMaEgZwaw54OMhbSGcKYpkheZzYizHZk01HewwZP_n7w3l58HESS_SEN9m7iKr-YwuNYNGCdFa4dhNxrcX2bwxKaHrTn0L9Qo11AsEl9EVBVkxsDliSgszitFUl_sET3zt8mv8g0iX1lAHVj0TXztM2HHHZkA3IWDhCSHlRyyvjUapG9IBkNe6yW2yg__YNegVeYGanE7p70L-l0cwXVE5lmXTBfZ-d-6NgorE_FI22leO5A

Create a NodePort and access the operator at https://kes-k8s-minio.lab.min.dev:30043 

kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "http","port": 9090,"protocol": "TCP","nodePort":31090}],"type": "NodePort"}}'
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "https","port": 9443,"protocol": "TCP","nodePort":30043}],"type": "NodePort"}}'

Show error - curl

curl http://192.168.64.68:31090/

Output

<!doctype html><html lang="en"><head><meta charset="utf-8"/><base href="/"/><meta content="width=device-width,initial-scale=1" name="viewport"/><meta content="#081C42" media="(prefers-color-scheme: light)" name="theme-color"/><meta content="#081C42" media="(prefers-color-scheme: dark)" name="theme-color"/><meta content="MinIO Console" name="description"/><meta name="minio-license" content="agpl" /><link href="./styles/root-styles.css" rel="stylesheet"/><link href="./apple-icon-180x180.png" rel="apple-touch-icon" sizes="180x180"/><link href="./favicon-32x32.png" rel="icon" sizes="32x32" type="image/png"/><link href="./favicon-96x96.png" rel="icon" sizes="96x96" type="image/png"/><link href="./favicon-16x16.png" rel="icon" sizes="16x16" type="image/png"/><link href="./manifest.json" rel="manifest"/><link color="#3a4e54" href="./safari-pinned-tab.svg" rel="mask-icon"/><title>MinIO Console</title><script defer="defer" src="./static/js/main.107f720c.js"></script><link href="./static/css/main.49948cf4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"><div id="preload"><img src="./images/background.svg"/> <img src="./images/background-wave-orig2.svg"/></div><div id="loader-block"><img src="./Loader.svg"/></div></div></body></html>

curl https://192.168.64.68:30043/

Output

curl: (7) Failed to connect to 192.168.64.68 port 30043 after 0 ms: Connection refused

Show error - logs

kubectl -n minio-operator get pods

Output

NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-86589f8868-b4hgv   0/1     Pending   0          85s
minio-operator-86589f8868-p2fzj   1/1     Running   0          85s
console-6d65b6d967-rs6w9          1/1     Running   0          70s

kubectl -n minio-operator logs pod/console-6d65b6d967-rs6w9

Output

E: 2023/11/14 21:04:58 Unable to load certs: unable to create certs CA directory at /tmp/certs/CAs: failed with mkdir /tmp/certs/CAs: read-only file system
Serving operator at http://[::]:9090

Workaround - Part 1

Fix

kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"volumes":[{"name": "cas", "emptyDir": {}}]}}}}'
kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"containers":[{"name": "console", "image": "minio/operator:v5.0.10", "volumeMounts":[{"name": "cas", "mountPath": "/tmp/certs/CAs"}]}]}}}}'

Show fix - logs

kubectl -n minio-operator get pods

Output

NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-86589f8868-b4hgv   0/1     Pending   0          8m37s
minio-operator-86589f8868-p2fzj   1/1     Running   0          8m37s
console-65fff7697f-4g59v          1/1     Running   0          33s

kubectl -n minio-operator logs pod/console-65fff7697f-4g59v

Output

Serving operator at http://[::]:9090

Workaround - Part 2

kubectl -n minio-operator get secrets

Output

NAME                TYPE                                  DATA   AGE
console-sa-secret   kubernetes.io/service-account-token   3      21m
console-tls         Opaque                                2      21m

kubectl -n minio-operator get deployment/console -o json | jq '.spec.template.spec.volumes[] | select( .name == "tls-certificates")' | jq '.projected.sources[0].secret.name'

Output

"operator-console-tls"

Fix

kubectl patch deployment -n minio-operator console -p '{"spec":{"template":{"spec":{"volumes":[{"name": "tls-certificates", "projected": {"sources": [{"secret": {"name": "console-tls"}}]}}]}}}}'

Show fix - logs

kubectl -n minio-operator get pods

Output

NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-86589f8868-b4hgv   0/1     Pending   0          9m59s
minio-operator-86589f8868-p2fzj   1/1     Running   0          9m59s
console-6c4f5ffb6d-zb9m2          1/1     Running   0          5s

kubectl -n minio-operator logs pod/console-6c4f5ffb6d-zb9m2

Output

Serving operator at http://[::]:9090
Serving operator at https://[::]:9443

Show fix with TLS enabled - curl

curl https://192.168.64.68:30043/ -k

Output

<!doctype html><html lang="en"><head><meta charset="utf-8"/><base href="/"/><meta content="width=device-width,initial-scale=1" name="viewport"/><meta content="#081C42" media="(prefers-color-scheme: light)" name="theme-color"/><meta content="#081C42" media="(prefers-color-scheme: dark)" name="theme-color"/><meta content="MinIO Console" name="description"/><meta name="minio-license" content="agpl" /><link href="./styles/root-styles.css" rel="stylesheet"/><link href="./apple-icon-180x180.png" rel="apple-touch-icon" sizes="180x180"/><link href="./favicon-32x32.png" rel="icon" sizes="32x32" type="image/png"/><link href="./favicon-96x96.png" rel="icon" sizes="96x96" type="image/png"/><link href="./favicon-16x16.png" rel="icon" sizes="16x16" type="image/png"/><link href="./manifest.json" rel="manifest"/><link color="#3a4e54" href="./safari-pinned-tab.svg" rel="mask-icon"/><title>MinIO Console</title><script defer="defer" src="./static/js/main.107f720c.js"></script><link href="./static/css/main.49948cf4.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"><div id="preload"><img src="./images/background.svg"/> <img src="./images/background-wave-orig2.svg"/></div><div id="loader-block"><img src="./Loader.svg"/></div></div></body></html>

Show version

kubectl -n minio-operator get pod/minio-operator-86589f8868-p2fzj -o json | jq '.spec.containers[] | select( .name == "minio-operator")' | jq '.image'

Output

"minio/operator:v5.0.10"
Clone this wiki locally