WIP - Add task prefixes to prometheus metrics

[mrocklin]

cc @ntabris

[ntabris]

Here's an example graph of the data in Grafana:

cc @ian-r-rose

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 40m 16s ⏱️ +23s
  3 136 tests +1    3 051 ✔️ +  3    85 💤 ±0  0 ❌  - 2 
23 206 runs  +7  22 300 ✔️ +11  906 💤  - 2  0 ❌  - 2 

Results for commit e02db9e. ± Comparison against base commit 8f36aa5.

[mrocklin]

Context for this:

We like the task stream plot, and we'd like some version of it that could be stored long-term. The task strream plot is a bit too busy. We have attempts of this with TaskGroups. Coiled has systems internally which does this. Maybe prometheus is better for this though. This PR samples the task stream and aggregagtes the information.

This is lossy, which isn't great. We could also use counters internally and do some sort of diff in grafana. @ntabris may think about this a bit.

[jacobtomlinson]

Neat!

[mrocklin]

This is superceded by #7088 , which gives a much more accurate view. This PR effectively samples the task stream while the other tracks all results using a bit of added state.

[hendrikmakait]

Superseded by #7088