Build out health monitoring

[Johennes]

Why?

We don’t have a good holistic overview of CI health outside of individual PRs. Some job failures are posted into rooms and some failure rates are available via external tools such as Cypress. There is, however, no way to get a holistic overview or monitor temporal evolution and things like job execution time are not tracked at all. This limits our ability to identify recurring issues and decreases confidence in the stability of our CI setup.

A related monitoring problem is that our Sentry instance is currently not very helpful, at times even called unusable. This prevents us from actually getting value out of this seemingly useful tool.

What?

We’ll build out a dashboard to monitor relevant CI health metrics such as failure rates and execution times. Additionally we’ll also make Sentry usable.

Plan

CI health monitoring

Beta Give feedback

1. Research available 3rd party tools
2. Research available APIs

Fix Sentry

Beta Give feedback

1. Document what's broken / inconvenient about Sentry today

Internal references

myhours: https://app.myhours.com/#/projects/2028386/overview

[paboum]

https://en.wikipedia.org/wiki/Worse_is_better

It seems you don't have time for this research. Just pick a random one, like https://github.com/otto-de/gitactionboard, and start using it ASAP, just monitor one thing but monitor it from today. This way it will already provide value from the day one and you can use the saved capacity for continuing the integration or shifting to another tool if you really can't continue with this one.

[Johennes]

Superseded by https://github.com/vector-im/wat-internal/issues/59 and https://github.com/vector-im/wat-internal/issues/60.