Skip to content
This repository has been archived by the owner on Oct 23, 2019. It is now read-only.

[questing] Empty grafana dashboard #219

Open
Igor-lkm opened this issue Aug 5, 2019 · 9 comments
Open

[questing] Empty grafana dashboard #219

Igor-lkm opened this issue Aug 5, 2019 · 9 comments

Comments

@Igor-lkm
Copy link

Igor-lkm commented Aug 5, 2019

Problem: Dashboard 1598 is empty, data from zipkin is in Prometheus

Setup: k8s cluster with zipkin, prometheus (helm) and grafana (helm)

I am using zipkin-transport-http to send data to zipkin

Dashboard: https://grafana.com/grafana/dashboards/1598

gnetId: 1598
revision: 15

Zipkin deployment config on k8s with Prometheus scraping configuration:

      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9411'
        prometheus.io/path: '/actuator/prometheus'
    spec:
      containers:
        - name: zipkin
          image: 'openzipkin/zipkin:2.15.0'

Example response produces by zipkin:9411/actuator/prometheus

zipkin_collector_messages_total{transport="http",} 223.0

Dashboard is empty:

Screenshot 2019-08-05 at 11 20 42

Data is in prometheus:

Screenshot 2019-08-05 at 11 20 58

Example: Messages received per transport

Example query from dashboard:

sum by(transport)(rate(zipkin_collector_messages_total{job="zipkin",instance=~"$instances"}[$__interval]))

And seems i do not have job="zipkin",instance=~"$instances"

So this would work ok:

sum by(transport)(rate(zipkin_collector_messages_total{}[5m]))

In table view looks like this:

Screenshot 2019-08-05 at 11 50 28

Job: kubernetes-pods

Seems somehow job and instance are missed 🤔 Maybe I am missing something?

Thanks for the help!

Added:

Works ok with different labels.

Looks like it is a broad question at the end.

For example RabbitMQ has metrics like rabbitmq_node_mem_used, prefixed with rabbitmq (dashboard 4279)

@codefromthecrypt
Copy link

codefromthecrypt commented Aug 7, 2019 via email

@mstaalesen
Copy link

Sorry for the late response (vacation).

Some of these parameters are based on what the labels are set for the scrape targets.
If I recall correctly, when I modified the dashboard, it relied on instance and job. So I just kept it like that.

So to solve this in your case @Igor-lkm, you need to change the query from jobs="zipkin" to jobs="kubernetes-pods
Instance is already present (shown in your last screenshot).

@Igor-lkm
Copy link
Author

@mstaalesen yep, thanks :)

So I did it:

Works ok with different labels.

The point was that it's not working out-of-the-box and if there is a way to make it work out-of-the-box 🤔

@mstaalesen
Copy link

Sorry, I missed that part!

One approach could be to just remove job="zipkin" from the queries. It should still work, as the zipkin-metrics have a rather unique names.

@Igor-lkm
Copy link
Author

Igor-lkm commented Aug 22, 2019

@mstaalesen If we remove job="zipkin", we would miss instances in the filter on top, as they depend on a job label:

  "label": "Zipkin instances",
  "multi": true,
  ...
  "query": "label_values(http_server_requests_seconds_bucket{job=\"zipkin\"}, instance)",

So seems to remove job="zipkin" is not an option 🤔

If we would be able to remove job="zipkin" and still get an instances list, that would solve the problem 🤔

@mstaalesen Is there a github repo with this board, so i can contribute if I would have a good idea? :)

@codefromthecrypt
Copy link

codefromthecrypt commented Aug 22, 2019 via email

@abesto
Copy link
Contributor

abesto commented Aug 23, 2019

there is no github repo. you can download and change it until we figure out
what normal is wrt change management of grafana dashboards

That's correct, I don't know of a standard way to properly version-control dashboards hosted on grafana.com. A quick search turns up https://git.abolivier.bzh/babolivier/grafana-dashboards-manager, but it's anybodies guess whether that actually adds value over "use a Git repo, and an admin sometimes syncs to grafana.com".

@codefromthecrypt
Copy link

codefromthecrypt commented Aug 23, 2019 via email

@mstaalesen
Copy link

  "query": "label_values(http_server_requests_seconds_bucket{job=\"zipkin\"}, instance)",

I think the "workaround" on that would be to just use a metric that we know is unique to zipkin, or if that doesn't work - add a regexp to filter out only zipkin instances.

With regards to how to version control it. There really is no good way of doing it as of right now. I think there is some work being done where you save your dashboard as code. I've seen some talks on the topic, but I don't think its mature enough to be used by "the masses" yet.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants