Grafana-k8s not rendering new dashboard from grafana-agent-k8s when the relation in between stays joined

[agileshaw]

We have a charm (tempest-k8s built on top of this patch) providing dashboard through granfa-agent-k8s to grafana-k8s. If we keep the relations between grafana-agent-k8s and grafana-k8s intact but deploy a new charm and relate it to grafana-agent-k8s:

$ juju deploy [charm]
$ juju relate [charm]:grafana-dashboard  grafana-agent-k8s:grafana-dashboards-consumer

we will not see the the dashboard showing on Grafana WebUI.

We can confirm that the dashboard file did land in grafana container and can be found by running the following command:

$ kubectl exec -n observability grafana-0 -it -c grafana -- /bin/bash -c "grep 'Tempest' /etc/grafana/provisioning/dashboards/*"

Manually deleting grafana-k8s pod or remove + rejoin the relation between grafana-k8s and grafana-agent-k8s solves the issue.

[chanchiwai-ray]

I am hitting the same issue here, when deploying single node sunbeam (2023.2/edge channel) and enable observability plugin. This should load some dashboard from the charms in the plugin via grafana-agent.

Using the method mentioned by @agileshaw fix the issue

[chanchiwai-ray]

I seems to be able to reproduce the issue, consider the following situation:

openstack-exporter (provides the dashboards) <-> grafana-agent <-> grafana (via dashboars' relation)

if I refresh the openstack-exporter with updated dashboard content (but maybe the same dashboard title), then the original dashboards and the updated dashboard will be gone from the Grafana Web UI. And the pebble logs will have the following warning that seems related to this issue:

2024-04-09T06:59:48.859Z [grafana] logger=ngalert.multiorg.alertmanager t=2024-04-09T06:59:48.859163884Z level=info msg="starting MultiOrg Alertmanager"
2024-04-09T06:59:49.040Z [grafana] logger=provisioning.dashboard t=2024-04-09T06:59:49.039945479Z level=warn msg="the same UID is used more than once" orgId=1 uid=rYdddlPWk times=2 providers=[Default]
2024-04-09T06:59:49.040Z [grafana] logger=provisioning.dashboard t=2024-04-09T06:59:49.040454949Z level=warn msg="dashboard title is not unique in folder" orgId=1 title="System Resources" folderID=0 times=2 providers=[Default]
2024-04-09T06:59:49.040Z [grafana] logger=provisioning.dashboard t=2024-04-09T06:59:49.040511613Z level=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" provider=Default orgId=1
2024-04-09T06:59:49.279Z [grafana] logger=context userId=1 orgId=1 uname=admin t=2024-04-09T06:59:49.279754844Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=10.5.0.3 time_ms=2 duration=2.85149ms size=0 referer= handler=/api/live/ws
2024-04-09T06:59:54.052Z [grafana] logger=provisioning.dashboard type=file name=Default t=2024-04-09T06:59:54.052120078Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=Default file=/etc/grafana/provisioning/dashboards/juju_grafana-agent-k8s_89c8bfc.json folderId=0
2024-04-09T06:59:54.089Z [grafana] logger=provisioning.dashboard type=file name=Default t=2024-04-09T06:59:54.089564192Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=Default file=/etc/grafana/provisioning/dashboards/juju_grafana-agent-k8s_0def0c2.json folderId=0
2024-04-09T06:59:59.055Z [grafana] logger=provisioning.dashboard type=file name=Default t=2024-04-09T06:59:59.055582882Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=Default file=/etc/grafana/provisioning/dashboards/juju_grafana-agent-k8s_89c8bfc.json folderId=0
2024-04-09T06:59:59.077Z [grafana] logger=provisioning.dashboard type=file name=Default t=2024-04-09T06:59:59.077823193Z level=warn msg="Not saving new dashboard due to restricted database access" provisioner=Default file=/etc/grafana/provisioning/dashboards/juju_grafana-agent-k8s_0def0c2.json folderId=0

And the below are the differences of the dashboards file in Grafana unit

ubuntu@sunbeam01:~$ juju ssh -m observability --container grafana grafana/0 ls /etc/grafana/provisioning/dashboards -l
total 1060
-rw-r--r-- 1 root root    141 Apr  9 06:59 default.yaml
-rw-r--r-- 1 root root 183537 Apr  9 06:48 juju_alertmanager-k8s_e9224b0.json
-rw-r--r-- 1 root root  32393 Apr  9 06:48 juju_grafana-agent-k8s_07d0d0a.json
-rw-r--r-- 1 root root 310490 Apr  9 06:48 juju_grafana-agent-k8s_0def0c2.json
-rw-r--r-- 1 root root  35047 Apr  9 06:48 juju_grafana-agent-k8s_7df704b.json
-rw-r--r-- 1 root root  49603 Apr  9 06:48 juju_grafana-agent-k8s_ce9d705.json
-rw-r--r-- 1 root root 310490 Apr  9 06:48 juju_grafana-agent_0def0c2.json
-rw-r--r-- 1 root root  33505 Apr  9 06:48 juju_grafana-agent_1807957.json
-rw-r--r-- 1 root root  57798 Apr  9 06:48 juju_loki-k8s_0804127.json
-rw-r--r-- 1 root root  52801 Apr  9 06:48 juju_prometheus-k8s_35dd368.json
ubuntu@sunbeam01:~$ juju ssh -m observability --container grafana grafana/0 ls /etc/grafana/provisioning/dashboards -l
total 1060
-rw-r--r-- 1 root root    141 Apr  9 06:59 default.yaml
-rw-r--r-- 1 root root 183537 Apr  9 06:59 juju_alertmanager-k8s_e9224b0.json
-rw-r--r-- 1 root root  32393 Apr  9 06:59 juju_grafana-agent-k8s_07d0d0a.json
-rw-r--r-- 1 root root 310490 Apr  9 06:59 juju_grafana-agent-k8s_0def0c2.json
-rw-r--r-- 1 root root  35047 Apr  9 06:59 juju_grafana-agent-k8s_7df704b.json
-rw-r--r-- 1 root root  49404 Apr  9 06:59 juju_grafana-agent-k8s_89c8bfc.json
-rw-r--r-- 1 root root 310490 Apr  9 06:59 juju_grafana-agent_0def0c2.json
-rw-r--r-- 1 root root  33505 Apr  9 06:59 juju_grafana-agent_1807957.json
-rw-r--r-- 1 root root  57798 Apr  9 06:59 juju_loki-k8s_0804127.json
-rw-r--r-- 1 root root  52801 Apr  9 06:59 juju_prometheus-k8s_35dd368.json

[samuelallan72]

Confirming that this is still an issue, that we can reproduce 100% of the time. It causes problems when refreshing charms that provide grafana dashboards, because then the dashboards don't update. We do have workarounds, but it's cumbersome:

• bouncing the relation between grafana-agent and grafana (dashboards endpoint)
• or restart grafana: juju ssh -m observability --container grafana grafana/0 /charm/bin/pebble restart grafana

[lucabello]

I tried multiple times but cannot reproduce this with:

grafana-agent-k8s (rev89)
grafana-k8s (rev118)

Could you try reproducing again with these revisions?

[simskij]

@samuelallan72 can you help us confirm what Luca said?

[samuelallan72]

@lucabello @simskij sure, I can take a look when I have time next. Thanks for looking into this! 🙂

Note that Ray and I were reproducing this with grafana-agent charm (not grafana-agent-k8s). Not sure if that makes a difference though, since the issue seemed to be in grafana-k8s.

[lucabello]

I'm closing in the meantime, but please feel free to reopen if that's still an issue :)

[samuelallan72]

grafana-agent-k8s (rev89)
grafana-k8s (rev118)

Could you try reproducing again with these revisions?

I tested this today with these revisions and can confirm that the issue is no longer present. :)