[prometheus-community/kube-prometheus-stack] error after installation service "kube-prometheus-stack-operator" not found, nodeexporter pod crash

[kristijorgji]

Describe the bug
Error right after executing helm install, for service node found and nodeexporter pod stuck in crashloop

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
prometheus-community/kube-prometheus-stack

Which version of the chart:
version: 13.10.0

What happened:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post "https://kube-prometheus-stack-operator.prometheus.svc:443/admission-prometheusrules/validate?timeout=10s": service "kube-prometheus-stack-operator" not found

Also the pod is in invalid state for nodeexporter

pod/prometheus-prometheus-node-exporter-vfgvq                0/1     CrashLoopBackOff   5          4m

kubectl -n prometheus logs -f pod/prometheus-prometheus-node-exporter-vfgvq -p
failed to try resolving symlinks in path "/var/log/pods/prometheus_prometheus-prometheus-node-exporter-vfgvq_894d04f6-0116-4cc7-b180-af9110dda5c2/node-exporter/6.log": lstat /var/log/pods/prometheus_prometheus-prometheus-node-exporter-vfgvq_894d04f6-0116-4cc7-b180-af9110dda5c2/node-exporter/6.log: no such file or directory

What you expected to happen:
Not be shown the error

Also the pod is in crashloopbackoff for node exporter

pod/prometheus-prometheus-node-exporter-vfgvq                0/1     CrashLoopBackOff   5          4m

How to reproduce it (as minimally and precisely as possible):

$ kubectl create namespace prometheus
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack --namespace prometheus

Changed values of values.yaml (only put values which differ from the defaults):
Nothing changed expect provide a namespace --namespace prometheus

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[maiorano84]

/reopen

This issue is also confirmed on my end. It appears to be a common problem for Windows 10 users. I was able to get around this by setting hostRootFsMount to false, but I don't know what the far-reaching consequences might be:

helm install prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount=false --set grafana.adminPassword=secret

[demisx]

Getting the same error on Mac OS Monterey v12.2.1/Docker Desktop and prometheus kube stack helm chart v33.1.0.

This is what I had to add to my local skaffold.yaml to make it work:

overrides:
  prometheus-node-exporter:
    hostRootFsMount:
      enabled: false