Add graphs for cluster violations to Grafana dashboard

[mclasmeier]

[stehessel]

Could you please save the dashboard under a new name (copy will be temporary) so it's easier to review the dashboard changes?

[mclasmeier]

Saved it additionally as RHACS Dataplane - Cluster Metrics (+ Cluster Violations).

[stehessel]

In general, could you add some information to the panel on what to do with this data? What is the expected / normal behavior? Do we expect there to always be zero violations, or are some violations acceptable? How could I get more information on what type of violations these are if I want to follow up?

[mclasmeier]

Preview dashboard updated: https://grafana-route-rhacs-observability.apps.acs-int-us-01.isbr.p1.openshiftapps.com/d/de195dda-1237-4285-af32-1fe0ad3d2ff1/57f53fcb-ad58-5fc3-8113-d8a7ec83161a?orgId=1

[mclasmeier]

In general, could you add some information to the panel on what to do with this data? What is the expected / normal behavior? Do we expect there to always be zero violations, or are some violations acceptable? How could I get more information on what type of violations these are if I want to follow up?

Where should I add these information? From a quick search on the dashboards I haven't found panels including such a description @stehessel.

[stehessel]

You can either put a text box next to the plots like so

or you can use the widget description. For longer text I would use the former.

For example, one might wonder how get to the component that generated the violation one sees on the dashboard. You can also just put a link to some other place where this info is available (e.g. a SOP or something like that).

[mclasmeier]

@stehessel Feel free to re-review. Descriptions for the panels will be added in a follow-up commit as part of this PR.

[mclasmeier]

Dashboard updated (https://grafana-route-rhacs-observability.apps.acs-int-us-01.isbr.p1.openshiftapps.com/d/de195dda-1237-4285-af32-1fe0ad3d2ff1/57f53fcb-ad58-5fc3-8113-d8a7ec83161a?orgId=1), now includes descriptions.

[stehessel]

LGTM thanks 🚀 . For extra credit add a link to CloudWatch :).

[mclasmeier]

Final changes -- board is updated: https://grafana-route-rhacs-observability.apps.acs-int-us-01.isbr.p1.openshiftapps.com/d/de195dda-1237-4285-af32-1fe0ad3d2ff1/57f53fcb-ad58-5fc3-8113-d8a7ec83161a?orgId=1&from=now-24h&to=now

Changes:

• Added CloudWatch Log Insight links
• Removed a bogus "axisWidth" setting, which caused the y-axis (specifically its labels) to be mostly invisible for the first graph.

[mclasmeier]

@stehessel Given the approval, I will proceed with merging. Besides having included the cloudwatch links I also fixed a minor inconsistency by removing a bogus axisWidth setting in one of the graphs. Thanks for the feedback!