Prometheus metrics experiment: Sharding and receiver monitoring

[dashpole]

Note: This is a pull request to my fork, not to upstream. For demo purposes only.

This takes a similar-ish approach as the prometheus operator did in: prometheus-operator/prometheus-operator#3241. This is simpler in that we do not have replicas (no HA). It also doesn't work with autoscalers, since the number of shards has to be manually adjusted (this doesn't include an operator to do that for us).

This implements sharding for the prometheus receiver using the hashmod function by injecting it into the prometheus configuration used in the receiver.

As inputs, it takes a total number of shards, and the pod name. The pod name can be obtained through the downward API. The shards should be equal to replicas, but we don't have an operator to enforce that, so you would have to manually update it.

The changes to the statefulset are in https://github.com/dashpole/dashpole_demos/blob/master/otel/collector/overlays/sharded_prometheus/statefulset.yaml#L20, and the changes to the otel configuration are https://github.com/dashpole/dashpole_demos/blob/master/otel/collector/overlays/sharded_prometheus/otel-collector-config.yaml#L8

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.