-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[prometheus-kube-stack] grafana not showing resource utilization (no data) #576
Comments
@nimamahmoudi The |
@stylianosrigas Thanks for pointing that out, I just followed the guidelines shown for issues. Here is the output of
|
I can confirm this bug as well and the updated query fixes the dashboards. |
Am also getting the same issues after installing on Kubernetes v1.20. (v1.20.2+k3s1) |
I can confirm this issue too |
I'm looking at this same issue in my environment, in a k3s 1.20 cluster like @stephenpope. What Kubernetes version/distribution are others running with this problem? The missing metrics seem to be container-level metrics, while the pod-level ones work fine. Could this be a kube-state-metrics 1.9 incompatibility with Kubernetes 1.20? I might try deploying the latest kube-state-metrics v2 beta and see if it helps at all... |
Same version here. The metrics don't look like they're missing, they're just here |
The default value is 1, @nimamahmoudi quite rightly said we're seeing this with a default helm install |
I've just rebuilt my k3s cluster on the latest version of k3s 1.20 and with the latest version of kube-prometheus-stack and this problem has magically vanished. This is obviously a really bad test and I have no idea which change resolved it for me, but has anyone else also seen this start working again? |
Allegedly this fixes it k3s-io/k3s#2831 @cablespaghetti |
Great! So this was a k3s bug all along. Makes sense! |
Was this fixed by something identified by k3s that wasn't specific to k3s? I was using stock Kubernetes 1.18 provided by EKS (with Prometheus sharding turned on) |
Yes @sidewinder12s, that's the running theory kubernetes/kubernetes#97006 (comment) |
I have same issues on kOps (k8s v1.19.7) and seems like issue is here: |
Please check out #809, that's what turned out to be the issue in my case, on a sharded/replicated prometheus with a Thanos install. |
This problem has reappeared for me on a new cluster. |
@ChipWolf I am currently experiencing this issue any fix |
No idea, it disappeared for a little while then re-appeared in a recent update |
I am having the same issue. Query update helps, but in case I have a lot of them that have to be passed to other users, are there any mechanisms to update it automatically? I described the issue in-detail here: grafana/grafana#50392 |
The same issue appears with EKS Kubernetes 1.21 and version 36.0.3 |
I have the same issue with k3s 1.25.0 and version v40.0.0 |
works for me with kind v1.24 and kube-prometheus-stack v40.0.0 |
For EKS 1.23 kube-prometheus-stack 40.1.0, the issue was kubelet's serviceMonitor running with |
This is still an issue with kube-prometheus-stack 40.1.0 on Kubernetes version v1.24.3. Cannot see the memory,cpu data associated with containers in Grafana dashboard. |
I got it working with the |
This is still an issue with kube-prometheus-stack 43.2.1 on Kubernetes version v1.24.4. Cannot see the memory,cpu data associated with containers in Grafana dashboard.
changed values
Any ideas how to fix this? |
Describe the bug
After installing the stack with default values, the resource metrics for pods don't show up ("No data" message shown on grafana).
Version of Helm and Kubernetes:
Helm Version:
Kubernetes Version:
Which chart:
prometheus-community/kube-prometheus-stack
Which version of the chart:
12.11.3
What happened:
I installed the prometheus operator using the following command on k3s:
What you expected to happen:
I expected the values to show up.
How to reproduce it (as minimally and precisely as possible):
Changed values of values.yaml (only put values which differ from the defaults):
values.yaml
Not changes, default values used.
The helm command that you execute and failing/misfunctioning:
For example:
helm install prometheus prometheus-community/kube-prometheus-stack
Helm values set after installation/upgrade:
Anything else we need to know:
An example is the
Kubernetes / Compute Resources / Workload
dashboard, where the first panel has the following query:But it will work when we change it to following (only changed second line):
It seems like the name of the metrics are different from what is being stored in Prometheus.
The text was updated successfully, but these errors were encountered: