Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards #1235

kysrpex · 2024-06-17T10:38:34Z

Update Grafana to version 11 and switch from deprecated role cloudalchemy.grafana to the official Grafana role from the grafana.grafana collection. Update usegalaxy_eu.grafana_matrix_forwarder. Instead of disabling firewalld on the Grafana host, open the nginx ports.

Before merging this PR, the Grafana database must be migrated from SQLite to PostgresSQL. This is rather simple using pgloader and following this guide from Jamie Ly. The postgres database information can be found here. It can be accessed from stats, sn06 and maintenance (see usegalaxy-eu/infrastructure-playbook#1235).

Migration in a nutshell:

Make sure the grafana database exists in Postgres and the grafana user has all privileges on it. You may run reset.sql as postgres to do this ( ⚠️ this drops the grafana database).
Run schema.sql as grafana to create all tables Grafana v9.2.10 (c37dcaf0da) needs. Grafana v11 will run database migrations on top of it.
Install pg_loader. Luckily, there are packages available in the Postgres community repository. To enable the repo, you just need the right repository package. You may (and should) delete the repository package once you have completed this migration.
Open grafana.load and adjust the path to /data/monitoring/grafana_data/grafana.db (if running the migration steps on stats.galaxyproject.eu) and the PostgreSQL connection string if needed.
Run pgloader grafana.load. Warnings and even errors are expected, not an issue.
Merge this PR and run the grafana.yml playbook.
Enable the stats-grafana Jenkins project.
Some dashboards (the provisioned ones) will be in the wrong folder. Move them to the correct folder (see pictures below). All the remaining dashboards that do not belong to an existing folder should be moved to a new folder "General".

Closes usegalaxy-eu/issues#558.

Update Grafana to version 11 and switch from deprecated role `cloudalchemy.grafana` to the official Grafana role from the `grafana.grafana` collection. Update `usegalaxy_eu.grafana_matrix_forwarder`. Instead of disabling firewalld on the Grafana host, open the nginx ports.

Add usegalaxy-eu/grafana-dashboards as a submodule so that the Ansible role `grafana.grafana.grafana` can provision the dashboards from this folder.

Grafana dashboards will be imported to Grafana's database and loaded from it rather than provisioned from a folder.

Datasource uids must be provided so that the imported dashboards can reference them

This reverts commit e2f4e87.

mira-miracoli · 2024-07-15T14:22:42Z

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "mounts"]
 	path = mounts
 	url = https://github.com/usegalaxy-eu/mounts
+[submodule "files/grafana"]
+	path = files/grafana
+	url = https://github.com/usegalaxy-eu/grafana-dashboards.git


I am not sure if the .git could lead to problems, but I am also not an expert, was just noticing, that mount doesn't have it

Should be fine afaik, that's even what GitHub suggests

ok :)
I just saw it at git handbook without and was not sure about this

group_vars/grafana/vars.yml

mira-miracoli · 2024-07-15T14:36:35Z

group_vars/grafana/vars.yml

+grafana_matrix_forwarder_port: 6000
+grafana_matrix_forwarder_username: "{{ vault_grafana_matrix_forwarder_username }}"
+grafana_matrix_forwarder_password: "{{ vault_grafana_matrix_forwarder_password }}"
+grafana_matrix_forwarder_resolve_mode: reaction


Thanks for updating / improving this!

mira-miracoli

Thank you!

bgruening · 2024-07-16T06:38:35Z

Cool beans! Thanks a lot!

I have a technical question, but please deploy.
Old links to dashboards will probably not work anymore, I assume, is there a way in grafana to keep stable links? Or should we use bitly or even better gx.io to create stable links dashboards?

kysrpex · 2024-07-16T07:06:17Z

Cool beans! Thanks a lot!

I have a technical question, but please deploy. Old links to dashboards will probably not work anymore, I assume, is there a way in grafana to keep stable links? Or should we use bitly or even better gx.io to create stable links dashboards?

With this migration path all dashboards keep the same uids, and Grafana has not changed how it forms the urls, so links do not break (see screenshots below).

bgruening · 2024-07-16T07:48:28Z

Oh nice!!!

bgruening · 2024-07-16T07:49:09Z

Can you please also sum that all up in a nice blog post and in addition in a operations update.

Thanks a lot!

Workaround for certbot error below. ``` The server will not issue certificates for the identifier :: Invalid identifiers requested :: Cannot issue for "stats-galaxyproject-eu.novalocal": Domain name does not end with a valid public suffix (TLD)", "Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details. ```

mira-miracoli · 2024-07-23T13:42:46Z

oh I guess my github was stuck

kysrpex · 2024-07-23T14:05:09Z

@sanjaysrikakulam @mira-miracoli Important detail. You may find that some panels do not work. You can very easily fix them like this:

Find a panel that does not work.
Click "Edit".
Click the pencil on the bottom right (raw query mode). The preview will start working instantly.
Save the dashboard (the screenshot is wrong, use "Save", not "Apply").

Don't ask me why this works. Actually if you switch the panel back to visual editor mode it breaks again.

bgruening · 2024-07-23T14:49:24Z

This one https://stats.galaxyproject.eu/d/000000004/galaxy?orgId=1&refresh=10s&viewPanel=7 does not seem to work. Maybe wrong host?

kysrpex · 2024-07-24T12:46:12Z

This one https://stats.galaxyproject.eu/d/000000004/galaxy?orgId=1&refresh=10s&viewPanel=7 does not seem to work. Maybe wrong host?

The list of hosts is populated by the InfluxQL query SHOW TAG VALUES FROM "cluster.queue" WITH KEY = "host", but the only possible outcome of that query right now is "maintenance.galaxyproject.eu", see the InfluxDB measurement query below.

> SELECT * FROM "cluster.queue" ORDER BY time desc LIMIT 10
name: cluster.queue
time                 count engine host                         schedd                state
----                 ----- ------ ----                         ------                -----
2024-07-24T12:38:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu suspended
2024-07-24T12:38:00Z 539   condor maintenance.galaxyproject.eu sn06.galaxyproject.eu running
2024-07-24T12:38:00Z 426   condor maintenance.galaxyproject.eu sn06.galaxyproject.eu idle
2024-07-24T12:38:00Z 2     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu held
2024-07-24T12:38:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu removed
2024-07-24T12:38:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu completed
2024-07-24T12:37:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu removed
2024-07-24T12:37:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu completed
2024-07-24T12:37:00Z 0     condor maintenance.galaxyproject.eu sn06.galaxyproject.eu suspended
2024-07-24T12:37:00Z 543   condor maintenance.galaxyproject.eu sn06.galaxyproject.eu running

Changing to SHOW TAG VALUES FROM "cluster.queue" WITH KEY = "schedd" allows choosing sn06.galaxyproject.eu too, and then the route timings work.

This more likely has to do with the setup of the maintenance node and the HTCondor migration than with the Grafana update.

Note to all: if you see fixable problems like these on the dashboards go ahead and fix them.

kysrpex added the enhancement label Jun 17, 2024

kysrpex self-assigned this Jun 17, 2024

kysrpex force-pushed the grafana_update_version_11 branch from 33ddf43 to acc9b9a Compare June 17, 2024 10:41

Provision dashboards from usegalaxy-eu/grafana-dashboards

ec5391c

Add usegalaxy-eu/grafana-dashboards as a submodule so that the Ansible role `grafana.grafana.grafana` can provision the dashboards from this folder.

kysrpex force-pushed the grafana_update_version_11 branch from 8de8a08 to ec5391c Compare June 17, 2024 12:50

kysrpex changed the title ~~Update Grafana to version 11~~ Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards Jun 17, 2024

kysrpex mentioned this pull request Jun 18, 2024

Add GitHub workflow that updates repositories where this repository is used as Git submodule usegalaxy-eu/grafana-dashboards#6

Closed

Move grafana group vars to a dedicated group folder

dea78ea

kysrpex force-pushed the grafana_update_version_11 branch 2 times, most recently from 18c801b to 514808d Compare June 21, 2024 13:47

kysrpex added 6 commits June 24, 2024 14:02

Move grafana secret group vars to the dedicated group folder

df0d1d8

Import Grafana dashboards instead of provisioning them

e2f4e87

Grafana dashboards will be imported to Grafana's database and loaded from it rather than provisioned from a folder.

Persist Grafana's data to dedicated PostgreSQL database

40585ae

Use useGalaxy.eu's fork of the grafana.grafana collection

001aed8

Enter datasource passwords correctly

e859036

Enter datasource uids

bbe09a8

Datasource uids must be provided so that the imported dashboards can reference them

kysrpex force-pushed the grafana_update_version_11 branch from 514808d to bbe09a8 Compare July 3, 2024 10:30

Revert "Import Grafana dashboards instead of provisioning them"

00e8fc1

This reverts commit e2f4e87.

kysrpex force-pushed the grafana_update_version_11 branch from 365c7b4 to 00e8fc1 Compare July 15, 2024 13:29

kysrpex marked this pull request as ready for review July 15, 2024 13:30

kysrpex mentioned this pull request Jul 15, 2024

Switch stats instance to "public-extended" network usegalaxy-eu/infrastructure#200

Merged

mira-miracoli reviewed Jul 15, 2024

View reviewed changes

mira-miracoli approved these changes Jul 16, 2024

View reviewed changes

Merge branch 'master' into grafana_update_version_11

4b8ecb6

kysrpex merged commit a22f19d into master Jul 23, 2024
2 checks passed

kysrpex deleted the grafana_update_version_11 branch July 23, 2024 13:42

mira-miracoli approved these changes Jul 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards #1235

Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards #1235

kysrpex commented Jun 17, 2024 •

edited

Loading

mira-miracoli Jul 15, 2024

kysrpex Jul 15, 2024

mira-miracoli Jul 15, 2024

mira-miracoli Jul 15, 2024

mira-miracoli left a comment

bgruening commented Jul 16, 2024

kysrpex commented Jul 16, 2024

bgruening commented Jul 16, 2024

bgruening commented Jul 16, 2024

mira-miracoli commented Jul 23, 2024

kysrpex commented Jul 23, 2024 •

edited

Loading

bgruening commented Jul 23, 2024

kysrpex commented Jul 24, 2024

Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards #1235

Update Grafana to version 11 and provision dashboards from usegalaxy-eu/grafana-dashboards #1235

Conversation

kysrpex commented Jun 17, 2024 • edited Loading

mira-miracoli Jul 15, 2024

Choose a reason for hiding this comment

kysrpex Jul 15, 2024

Choose a reason for hiding this comment

mira-miracoli Jul 15, 2024

Choose a reason for hiding this comment

mira-miracoli Jul 15, 2024

Choose a reason for hiding this comment

mira-miracoli left a comment

Choose a reason for hiding this comment

bgruening commented Jul 16, 2024

kysrpex commented Jul 16, 2024

bgruening commented Jul 16, 2024

bgruening commented Jul 16, 2024

mira-miracoli commented Jul 23, 2024

kysrpex commented Jul 23, 2024 • edited Loading

bgruening commented Jul 23, 2024

kysrpex commented Jul 24, 2024

kysrpex commented Jun 17, 2024 •

edited

Loading

kysrpex commented Jul 23, 2024 •

edited

Loading