Track channel utilization using metrics

[huitseeker]

We're about to merge #637 which tracks channel capacity by logging it every 5s.

We'd rather take inspiration from https://github.com/diem/diem/blob/latest/common/channel/src/lib.rs, which creates IntGauge instances per channel for the occupancy of their butter. Note this channel wrapper is based on the std::sync::mpsc, while we rather use tokio::sync::mpsc.

In order to the integrate this in our metrics, we could:

1. Add the IntGauge metric in the PrimaryMetrics
2. We initialize and retrieve the metrics already here (channels will need to be initialised after the metrics)
3. Then we do have two options (a) pass the PrimaryMetrics in the new method (b) we just clone and pass the gauge directly. We probably want to do the (a) for consistency

We could also use #637 but just emitting the metrics in gauge. And we’ll need a label with the channel name there so we can differentiate the values.

(thanks to @akichidis for a lot of that direction)