💡 [Feature] Add recording rules to Prometheus configuration to store hourly/daily metrics

[huard]

... trying to split the metrics collection issue into small digestible bits.

Description

1. Select among all existing metrics recorded by Prometheus those of interest to external stakeholders:

• Number of active users
• CPU usage
• Bandwidth usage
• ...

2. Create recording rules in our current Prometheus server to store hourly/daily resolution metrics:

• Total number of active users per day
• Mean hourly CPU usage
• Mean hourly bandwidth usage
• ...

Make sure the frequency is clearly indicated in the metrics names.

3. Display hourly/daily data in Graphana dashboard

This doesn't solve the data retention issue, but it's a necessary step. A second Prometheus instance can federate the first instance and scrape the metrics that match a hourly/daily regexp pattern.

References

#277

Concerned Organizations

[mishaschwartz]

@huard you mentioned that there is an existing metric that records total number of active users per day. Do you know which metric that is (its name, how it is collected)?

[huard]

The only thing I know is that it's part of the vanilla config that came with the docker image. @tlvu would know more about this.

[tlvu]

The only thing I know is that it's part of the vanilla config that came with the docker image. @tlvu would know more about this.

No we do not have that metric (total number of active users per day).

All the metrics that comes with the "vanilla" config are listed here https://github.com/bird-house/birdhouse-deploy/blob/9d9f46c497e2b00a6ad5be9f1e3ec322f85868a3/birdhouse/components/README.rst#grafana-dashboard