Add alert rules to argo-controller based on the KF093 spec

[rgildein]

These alert rules provide an overview of all service states.

Using the KubeflowServiceDown or KubeflowServiceIsNotStable filter, the user
can easily see the status of all Kubeflow services.

These changes can be tested by running the following commands:

tox -e integration -- --keep-models
juju -m <model-name> show-unit grafana-agent-k8s/0 --endpoint metrics-endpoint | yq '.[]."relation-info".[]."application-data".alert_rules | fromjson'
# if you have cos deployed
juju consume <controller>:cos.remote-write
juju integrate remote-write grafana-agent-k8s
# open Grafana UI and check if KubeflowServiceDown and KubeflowServiceIsNotStable are there
# you can even stop pebble service to see alert rules firing
kubectl exec -it -n <model-name> pod/<pod-name> -c <workload-container> -- /charm/bin/pebble start <service-name>

part-of: #1026

[misohu]

I think we are missing tests for this change. I believe the in the issue canonical/bundle-kubeflow#1026 we are talking about using this https://github.com/canonical/charmed-kubeflow-chisme/blob/b64f6ccca228d08c6533cbe83382306cd46670dc/src/charmed_kubeflow_chisme/testing/cos_integration.py#L413 for testing.

Can you also please provide minimal steps for reviewer to test it locally? What should I deploy and what should I check.

[rgildein]

I think we are missing tests for this change. I believe the in the issue canonical/bundle-kubeflow#1026 we are talking about using this https://github.com/canonical/charmed-kubeflow-chisme/blob/b64f6ccca228d08c6533cbe83382306cd46670dc/src/charmed_kubeflow_chisme/testing/cos_integration.py#L413 for testing.

That's the recommended way. I created this and other PRs like this by script, so I did not check if charm implement the testing or not. That means that for some charms (like this one) we only update the alert rule and for some we add it. Do you think that we should include those test to all of them?

Can you also please provide minimal steps for reviewer to test it locally? What should I deploy and what should I check.

Something like deploy grafana-agent and check that alert rule is part of relation or deploy whole cos and check it in Grafana UI? WDYT?

[misohu]

LGTM :)