No metrics with 5.3.0-latest.2021125.2

[georok]

Describe the bug
After installing the 5.3.0-latest.2021125.2 version (both on Windows 10 and Rocky Linux 8.5) the metrics for GKE based cluster are not available.

To Reproduce
Steps to reproduce the behavior:

1. Install 5.2.7 - the metrics are available
2. Install 5.3.0 - the metrics are not available
3. Uninstall 5.3.0 and install 5.2.7 again - metrics are available again

Expected behavior
The metric should be availabe

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Lens Version: 5.3.0-latest.2021125.2
• OS: Windows 10 and Rocky Linux 8.5 (RHEL 8.5)
• Installation method (e.g. snap or AppImage in Linux): Windows - exe; Rocky - rpm
• GKE base cluster
• Lens metrics configuration: tried Auto detect (works on 5.2.7) and Prometheus Operator (works on 5.2.7)

Kubeconfig:
5.2.7 works with the exact same Kubeconfig, so the latter is not an issue

Additional context
Add any other context about the problem here.

[jim-docker]

I have noticed a delay, you may have to wait a few minutes for the node CPU/Memory metrics to come up.

[georok]

Of course. I noticed that with the previous versions. I switched back to 5.2.7 (sorry I noticed I mistyped up in the explanation and put 4.2.7) on Windows 10, because I have work to do. But I left my Linux version up for hours and still no metrics

[jim-docker]

OK, thanks. Can you check if this is true for other clusters to see if it is specific to your GKE cluster? The metrics work for me on linux and windows for various clusters

[georok]

All clusters (3) we have are GKE. Does not work on any of them. I installed 5.30 on Win 10 again and left it running for 30 minutes (so far) and still no metrics.
I have a K3s as personal playground, but I do not have the Prometheus stack on it. I can give it a try ...

[georok]

Now, that is interesting. I fired up my K3s cluster and Lens 5.3.0 recognizes that there is no Prometheus installation:

So, with our GKE clusters it recognizes that there is a Prometheus deployment, but the "metrics are not available at the moment". How can I interpret that message? Where can I look for any hints? Where are the logs located in Win 10? thanks

[georok]

After I installed Prometheus on the K3s, Lens started showing the same message as with the GKE clusters:

I installed the Prometheus stack this way:
helm install pcore prometheus-community/kube-prometheus-stack -n monitoring

[georok]

I uninstalled Lens v.5.3.0 and installed v.5.2.7 and the metrics are back immediately:

[jim-docker]

Thanks for the further details. For logs: https://docs.k8slens.dev/main/faq/#where-can-i-find-application-logs

If you can try one more thing, go to the Cluster Settings -> Metrics and set the Prometheus option to Helm (assuming it is currently Auto detect). Shouldn't take more than 2 minutes for the metrics to appear (the problem is confirmed otherwise)

[georok]

If I set the Prometheus option to Helm it it does not recognize it at all - as if Prometheus was never installed. I tried with both with empty Prometheus service address and with a hard-coded one (monitoring/pcore-kube-prometheus-stac-prometheus:9090).
When I switch back to Prometheus Operator it just says that metrics are not available at the moment.

[jim-docker]

Sorry, didn't realize kube-prometheus-stack = prometheus operator. So neither auto detect nor prometheus operator give you full metrics. You've demonstrated it's clearly a 5.3 issue, we'll try to reproduce. Someone else has noticed an issue around this too: #4436

[georok]

The last piece of information that comes to my mind - Our clusters were @ kube-prometheus-stack-18.0.5. This is what I have on one of our prod servers:
helm -n monitoring list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
bbe monitoring 1 2021-09-08 13:46:23.315248782 -0400 EDT deployed prometheus-blackbox-exporter-5.0.3 0.19.0
pcore monitoring 1 2021-09-08 13:36:29.983738197 -0400 EDT deployed kube-prometheus-stack-18.0.5 0.50.0

Yesterday, I updated the Prometheus stack thinking that the older version could be causing the problem. This is what I have on our Dev server today:

helm -n monitoring list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
bbe monitoring 1 2021-08-30 15:14:37.876255211 -0400 EDT deployed prometheus-blackbox-exporter-5.0.3 0.19.0
pcore monitoring 2 2021-11-25 15:49:29.213003765 -0500 EST deployed kube-prometheus-stack-20.0.1 0.52.0

The Prometheus stack is @ kube-prometheus-stack-20.0.1, but the issue is still there.
Today I installed the latest version on my personal K3s and still the metrics were not available.

I hope this helps, thanks

[Mateus-Romera]

This is happening to me too. I'm using a EKS base cluster, kube-prometheus-stack (Auto detect) and Lens on AppImages. It works fine on 5.2.7, but on 5.3.0, only the memory metrics is available in the "Nodes" tab.

[mboutet]

Exact same issue using AKS (Azure):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"2b2f82eb285cfe136e312ea94d806520c86fb599", GitTreeState:"clean", BuildDate:"2021-11-01T16:42:12Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

I use the kube-prometheus-stack chart from Bitnami installed in the prometheus namespace. The prometheus-operator version is 0.50.0.

[bravecobra]

Where can we download previous versions? That way we can find the version where it broke...

[SDedik]

Also seeing this issue with Azure AKS and Prometheus installed on our clusters.
For those who wondering, direct download link to previous version for Windows is https://lens-binaries.s3-eu-west-1.amazonaws.com/ide/Lens%20Setup%205.2.7-latest.20211110.1.exe

[nderraugh]

Confirmed with microk8s distribution of k8s @ 1.22.3 with prom installed through the enable command which previously worked find with Lens.

[ghost]

Since Lens 5.3.0, same here with AKS, k8s 1.20.7. I'm only able to see memory metrics.
Using kube-prometheus-stack-19.3.0 with app version 0.50.0

[andyliddle]

Having the same issue with Lens-5.3.1-latest/20211130.1, we are using the kube-prometheus-stack-21.0.0 with app version 0.52.0

[Mateus-Romera]

I'm seeing same behavior on the new v5.3.2-latest.20211201.1

[bravecobra]

I actually downgraded for this particular bug. Any reason why this is being postponed?

[AndrewSav]

I'm also affected by this issue, looking forward for it to be fixed!

[georok]

It looks like for one reason or another this issue is not going to be resolved soon. Is there a way to tell Lens on Windows to stop checking for updates? Every morning when I start it it tells me that there is a newer version even though I click on the No button every time. It is annoying. Obviously, I am not going to upgrade from a working version to a broken one.

[Paragon1970]

@georok I just renamed the app-update.ymlfile and it appears to disable this check on Mac
It contains the following:

provider: s3
bucket: lens-binaries
path: /ide
channel: latest
updaterCacheDirName: lens-updater

$ cd /Applications/Lens.app/Contents/Resources
$ mv app-update.yml app-update.yml-orig

Just realised you specifically asked after windows version, sorry don't have that at hand.

[georok]

Thank you @Paragon1970. I found it on Windows. I will give it a try. thanks,

[AndrewSav]

5.2.7 for Windows: https://lens-binaries.s3-eu-west-1.amazonaws.com/ide/Lens%20Setup%205.2.7-latest.20211110.1.exe

[ldnvnbl]

Same issue, Lens: 5.3.3-latest.20211223.1, EKS

[vitaliyf]

I went looking around at https://github.com/lensapp/lens/blob/master/src/main/prometheus/operator.ts to see what metrics Lens is using, and found this:

• A month ago, when we were using prometheus chart we had metric data that looked like the following, and this was working with Lens 5.3.3:
  node_memory_MemTotal_bytes{app="prometheus", app_kubernetes_io_managed_by="Helm", chart="prometheus-14.11.1", component="node-exporter", heritage="Helm", instance="11.22.33.44:9100", job="kubernetes-service-endpoints", kubernetes_name="prometheus-node-exporter", kubernetes_namespace="monitoring", kubernetes_node="node-hostname-redacted.example.com", release="prometheus"}

• after we switched to kube-prometheus-stack (v24.x and prometheus-operator) we are now seeing the same metric with these labels (and they aren't showing up in Lens anymore):
  node_memory_MemTotal_bytes{container="node-exporter", endpoint="http-metrics", instance="11.22.33.44:9100", job="node-exporter", namespace="monitoring", pod="prometheus-prometheus-node-exporter-jk256", service="prometheus-prometheus-node-exporter"}

Notice how there is no longer a "node" label with the hostname, and this is why metrics aren't showing up in Lens. Similar issue exists for kube-state-metrics metrics.

As a workaround, you can add "node" label by adding a relabeling to both node-exporter and kube-state-metrics, something like in your kube-prometheus-stack's values:

kube-state-metrics:
  prometheus:
    monitor:
      enabled: true
      relabelings:
        - action: replace
          sourceLabels:
          - __meta_kubernetes_endpoint_node_name
          targetLabel: node

or patch existing ServiceMonitor objects for both to add similar:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
  - port: http-metrics
    relabelings:
    - action: replace
      sourceLabels:
      - __meta_kubernetes_endpoint_node_name
      targetLabel: node
    scheme: http

[jim-docker]

When installing kube-prometheus-stack using helm I had to do this before prometheus-node-exporter would run properly (and for the metrics to appear)

[headyj]

Still no metrics with 5.3.4 and latest node-exporter helm chart :(

[georok]

All looks good with 5.3.4 on Rocky Linux. The metrics are back for me. I will check the Windows version shortly.

[nevalla]

Still no metrics with 5.3.4 and latest node-exporter helm chart :(

node-exporter just allows prometheus to scrape node metrics. How did you install Prometheus?

[Paragon1970]

Can confirm v5.3.4 node metrics are being displayed on macOS.

Looks like the fix was sorted in this commit: 35c8b76

[bravecobra]

TLDR: check whether the metrics are in Prometheus in the first place.

I was having the same issue. However I dug a bit deeper in and found out that, as I was using k3d to spin my k3s nodes using memory limits, I had no values in Prometheus for the memory usage.

I'm using the Helm chart kube-prometheus-stack version 30.1.0

I was using the following command to spin up my local cluster

k3d cluster create ${each.key} --config=${local_file.cluster-config[each.key].filename} --servers-memory ${var.server-memory} --agents-memory ${var.agent-memory}

This implementation uses fake /proc/meminfo mounts which don't reflect usage. Since the Prometheus node exporters are reading that to report memory usage, the values didn't end up in Prometheus, hence no metrics in Lens either.

So first, make sure whether your nodes actually report back. On a node shell:

# egrep MemAvailable /proc/meminfo
MemAvailable:   xxxxx kB

Next you actually want that value in Prometheus. Look for node_memory_MemAvailable_bytes.
So after removing the --servers-memory and --agents-memory flags, I got my metrics back all the way up to Lens.

Now it works out of the box again without any additional configuration. I need to find another way to limit the memory of my k3s nodes.

[AndrewSav]

I can see metrics again now in 5.4.3 (Windows)

[headyj]

Still no metrics with 5.3.4 and latest node-exporter helm chart :(

node-exporter just allows prometheus to scrape node metrics. How did you install Prometheus?

I installed each chart independently, but everything was working until I updated both Lens and Prometheus versions so I'm not quite sure which one is responsible.

I also checked from prometheus and node-exporter metrics themselves (like node_memory_MemAvailable_bytes) are available

EDIT: I triy to downgrade to 5.2.5 but it's still not working, so I suppose that this problem is related to the Prometheus release and not Lens itself

[AndreiPaulau]

Hello, 5.3.4 works well! Many thanks!

Gents, again, that was the complex issue. The one "let's say, issue" had to be adopted by lens, the second - 'prom stack' devs had to fix this issue as well (as @nevalla mentioned earlier) - prometheus-community/helm-charts#1631 (Or you may workaround this by your own as discussed in the thread)

Please ensure your values are correct.:

serviceMonitorSelector:
    matchLabels:

AFAIK the issue was found starting 25.x.y helm chart

[ljakab]

Seeing same issue on Macbook Lens v5.3.3 (no cpu, disk metrics in overview / nodes)

Linux Lens v5.0.0 showing full metrics (cpu, memory, disk) pointing to same cluster as above.

Nothing obvious jumps out from logs. DEBUG=TRUE MacOS: ~/Library/Logs/Lens/lens(x).log

Wild guess if I'm honest but could it be related to Cleanup Prometheus detection (#4316) commit? 5d172b8

Downgraded to 5.2.7. I had a bit of a job locating the 5.2.7 dmg image so posting here for others: https://lens-binaries.s3-eu-west-1.amazonaws.com/ide/Lens-5.2.7-latest.20211214.2.dmg

I get an access denied, do you still have this file somewhere?

[headyj]

Still no metrics with 5.3.4 and latest node-exporter helm chart :(

node-exporter just allows prometheus to scrape node metrics. How did you install Prometheus?

IMO, the problem is that I don't have "kubernetes_node" field by default in kubernetes-node-exporter:
https://github.com/lensapp/lens/blob/master/src/main/prometheus/lens.ts#L61

I only have "node", referencing the node name, and so the group by is not working

[headyj]

Yep, confirmed, I relabeled "node" to "kubernetes_node" and everything is back.

It has been changed on prometheus side 2 months ago. so I thing this should be adapted somehow on Lens side: prometheus-community/helm-charts@df8add6#diff-a7d2f872af425efb23e365d063a23bdfcc33de8728c2b7bb4d536b7b8e80981aL1564

[balakumarpg]

In which release this fix is included. Still getting the same issue at mac os Monterey 12.2.1 and Lens: 5.3.4-latest.20220120.1. And I could not find a way to install 5.2.7. As mentioned in previous message, the dmg is not downloadable, Access denied :-(. Kindly help.

[AndrewSav]

@balakumarpg the fix fixed a particular issue that was introduced after 5.2.7. If you have not used it prior to 5.2.7 and you are on the lastes version and now you are getting this error, it is most likely unrelated to this thread, and a problem with your prometheus configuration.

[georok]

It hasn't been fixed actually. After the latest "fix" was introduced in 5.3.4, the metrics appeared again when running against older versions of the kube-prometheus-stack. Over the weekend, I upgraded to the latest version (kube-prometheus-stack-32.2.0) and the metrics disappeared again. So, no, it has not been truly fixed. It does not always work out-of the-box.
To make it work I had to relabel as was suggested several times in this thread. I added the following lines to my custom values.yaml file and redeployed the kube-prometheus-stack Helm chart:
kube-state-metrics:
prometheus:
monitor:
enabled: true
relabelings:
- action: replace
sourceLabels:
- __meta_kubernetes_endpoint_node_name
targetLabel: node

prometheus-node-exporter:
prometheus:
monitor:
enabled: true
relabelings:
- action: replace
sourceLabels:
- __meta_kubernetes_endpoint_node_name
targetLabel: node

[Jeansen]

Still an issue in 5.4.1 with Prometheus 15.5.3. The relabeling above did not work for me. With the latest Prometheus 14.x (14.12.0), it works out of the box.

[discostu105]

Can confirm, this works with Prometheus 14.12.0. Helm chart install: helm upgrade --install prometheus prometheus-community/prometheus --namespace observability --create-namespace --version=14.12.0

[ypicard]

Using the following:

• kube-system-promotheus v35.2.0
• Lens v5.4.6-latest.20220428.1

Metrics were not showing out of the box.

I had to apply the instructions mentionned here for it to finally work.

Using helm, this requires the following updates when deploying the chart:

  - name: prometheus
    chart: prometheus-community/kube-prometheus-stack
    version: 35.2.0
    values:
      - prometheus-node-exporter:
          prometheus:
            monitor:
              metricRelabelings:
                - action: replace
                  regex: (.*)
                  replacement: $1
                  sourceLabels:
                    - __meta_kubernetes_pod_node_name
                  targetLabel: kubernetes_node
        kubelet:
          serviceMonitor:
            metricRelabelings:
              - action: replace
                sourceLabels:
                  - node
                targetLabel: instance

I do not really know the whys and hows this is required, but this has been a painful experience to fix, as it was working out of the box before. I hope this helps someone.

[Nokel81]

@ypicard The reason that this is required is that in version 35 of kube-prometheus-stack they changed the default labeling for __meta_kubernetes_pod_node_name to be just node where previously it was kubernetes_node. #3955 covers what is needed to solve this without having to change the Prometheus labelling rules.

[celvin]

did someone make it work with the latest Lens? :(

[besteban1989]

Using the following:

• kube-system-promotheus v35.2.0
• Lens v5.4.6-latest.20220428.1

Metrics were not showing out of the box.

I had to apply the instructions mentionned here for it to finally work.

Using helm, this requires the following updates when deploying the chart:

  - name: prometheus
    chart: prometheus-community/kube-prometheus-stack
    version: 35.2.0
    values:
      - prometheus-node-exporter:
          prometheus:
            monitor:
              metricRelabelings:
                - action: replace
                  regex: (.*)
                  replacement: $1
                  sourceLabels:
                    - __meta_kubernetes_pod_node_name
                  targetLabel: kubernetes_node
        kubelet:
          serviceMonitor:
            metricRelabelings:
              - action: replace
                sourceLabels:
                  - node
                targetLabel: instance

I do not really know the whys and hows this is required, but this has been a painful experience to fix, as it was working out of the box before. I hope this helps someone.

This saved my a$$ thanks a lot !! I used latest kube-prometheus-stack chart 56.8.2 (v0.71.2 app version)