-
Notifications
You must be signed in to change notification settings - Fork 171
Prometheus Node Exporter bug (pod crashes) workaround
Adao Junior edited this page Feb 24, 2022
·
1 revision
The current version of the prometheus node exporter has a bug, mainly when deploying on Docker Desktop Kubernetes, that prevents the pod to have the running status.
The reason is known bug. This wiki shows the work around for this.
After deploy MuShop utilities, and notice that the prometheus node exporter is crashing, looks like that:
mushop-utilities mushop-utils-cert-manager-6c98bfb945-c66bv 1/1 Running 0 4m17s
mushop-utilities mushop-utils-cert-manager-cainjector-7958f9b88-8xjcz 1/1 Running 0 4m17s
mushop-utilities mushop-utils-cert-manager-webhook-5d6c95596c-z98gv 1/1 Running 0 4m17s
mushop-utilities mushop-utils-grafana-587c55988b-jscv2 3/3 Running 0 4m17s
mushop-utilities mushop-utils-ingress-nginx-controller-66fc9f67dc-tn8kb 1/1 Running 0 4m17s
mushop-utilities mushop-utils-kube-state-metrics-88ddb4bf6-85dhg 1/1 Running 0 4m17s
mushop-utilities mushop-utils-metrics-server-5744d9b98d-xz7ll 1/1 Running 0 4m17s
mushop-utilities mushop-utils-prometheus-alertmanager-cc4d9f68c-bcg9d 2/2 Running 0 4m17s
mushop-utilities mushop-utils-prometheus-node-exporter-rdxmx 0/1 CrashLoopBackOff 5 (86s ago) 4m17s
mushop-utilities mushop-utils-prometheus-pushgateway-7df8bbd6fd-9wn7d 1/1 Running 0 4m17s
mushop-utilities mushop-utils-prometheus-server-cbf8fc4bc-fr4df 2/2 Running 0 4m17s
Run the command:
kubectl -n mushop-utilities patch ds mushop-utils-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'
This will fix the DaemonSet and the pod will have the status running.
This issue affects only if you used the code before February 24th/2022. After that the MuShop code includes the workaround on the deployment.