Grafana dashboard is off compared to metrics

[remimikalsen]

The problem:
This Grafana dashboard doesn't seem to be reporting right numbers for ServerBytes

• https://grafana.com/dashboards/2949

My efforts so far:
Can the problem be that the nginx-vts-exporter outputs metrics in scientific notation, which then is misinterpreted by Grafana? This seems plausible from my numbers. A quick test with Go shows that printing float64 defaults to scientific notation, whereas printing uint64 results in only digits. Hoewever, changing types in nginx_vts_exporter.go isn't compatible with the function signature for prometheus.MustNewConstMetric.

Is there some way to force formatting in the nginx-vts-exporter metrics output to check if that's the problem?

My Grafana dashboard shows the following:

• ServerBytes for the all hosts peak at 12.92KiB out
• ServerBytes for the owncloud host peak at 4.1KiB out
• The JSON-file and the nginx-vts-exporter metrics are accumulated numbers, but even so the grafana graph seems far off.

http://nginx-vts-exporter:9913/metrics contains, among other data, the following lines:

nginx_server_bytes{direction="in",host="my.owncloud.host"} 6.7403731e+07
nginx_server_bytes{direction="out",host="my.owncloud.host"} 6.8422731e+07
nginx_upstream_bytes{backend="172.22.0.7:8080",direction="in",upstream="my.owncloud.host"} 6.7403731e+07
nginx_upstream_bytes{backend="172.22.0.7:8080",direction="out",upstream="my.owncloud.host"} 6.8422731e+07

• I note that there are float64 numbers formatted in the scientific notation, wondering if that could be an issue when reading the value into Grafana
• I have cross checked the numbers, and they seem to add up when comparing with http://nginx-vts-status/status/format/json

My nginx-vts-exporter on docker-compose.yml:

    nginx-vts-exporter:
    container_name: nginx-vts-exporter
    image: sophos/nginx-vts-exporter
    restart: always
    networks:
      - backend
      - frontend
    environment:
      - TZ=Europe/Oslo
      - NGINX_STATUS=http://nginx-vts-status/status/format/json

From prometheus.yml scrape_configs (scrape interval 5 seconds):

    - job_name: 'nginx-vts-exporter:'
      static_configs:
        - targets: ['nginx-vts-exporter:9913']

The prometheus data source seems to be working well in Grafana; the node exporter f.ex. seems to report correctly.

Best Regards,
Remi

[remimikalsen]

When I try this now, it seems to work. I've double checked the results with queries directly in Prometheus, and things check more or less out.

One thing is still off though: visualizing a time range for which I don't have 100% data coverage. E.g. visualizing 24h, and I only have data for the last 18 hours. Then the Y-axis in my case is way off, just like I describe in the the original issue. I believe that the reason things work for me now is that a few hours have passed since I opened the issie, and the data coverage for my default range is complete, and therefore I get the results I expect.

So probably, my troubles were due to an edge case. It would be nice to have somone confirm it, though.