Document prow monitoring

Add short readme for the monitoring stack and link to it from the main
prow readme.

Signed-off-by: Lennart Jern <[email protected]>

prow/README.md

@@ -111,6 +111,10 @@ avoid deleting it even if the Service is deleted. See
 [cert-manager](https://cert-manager.io/) and the Let's Encrypt HTTP01 challenge,
 as seen in [infra/cluster-issuer-http.yaml](infra/cluster-issuer-http.yaml).
 
+#### Monitoring
+
+Please see [infra/kube-prometheus](infra/kube-prometheus).
+
 ## Building node images
 
 The Kubernetes cluster where Prow runs needs pre-built images for the Nodes. We

prow/infra/kube-prometheus/README.md

@@ -0,0 +1,33 @@
+# Kube-prometheus for Metal3 Prow
+
+This monitoring stack is based on
+[kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/tree/main).
+We also took inspiration from how [k8s.io is monitoring
+ProwJobs](https://github.com/kubernetes/k8s.io/pull/5355).
+
+This is how you apply it in the cluster:
+
+```bash
+kubectl apply -f manifests/setup
+kubectl apply -f manifests
+kubectl apply -f prow-rules.yaml
+```
+
+The `manifests` are rendered using jsonnet based on
+`metal3-kube-prometheus.jsonnet`. Use the build script to render them after
+making changes:
+
+```bash
+make build
+```
+
+## How to access?
+
+For now, we have not exposed grafana or any other component. You can access them
+by using port-forward like this (after setting up access to the cluster itself):
+
+```bash
+kubectl -n monitoring port-forward svc/grafana 3000
+```
+
+Then go to <localhost:3000>.