roachprod, roachtest: add grafana annotations API

[DarrylWong]

This change adds the ability to create Grafana annotations from both roachprod and roachtests. The API handles authentication as well as creating an HTTP client and request. See individual commits for more details.

Release note: none
Epic: none
Fixes: #119821

[cockroach-teamcity]

This change is 

[DarrylWong]

Last commit can be dropped if we don't think the annotations are helpful there. It was more of a proof of concept + way to test that it actually works.

Note: if you want to play around with it yourself, I've only added annotation queries to the sql dashboard so far.

---

I played around with using clusterID instead of the runID and it worked, but found an edge case where if we are passing in an already existing cluster (using roachtest --cluster), the clusterID will be reused for potentially the same test.

I also found another edge case: if we run the same test multiple times using --count. In that case, we have to know the testRunID (test name + run #) instead of just the test name. I don't see a way to get around this besides exposing the testRunID as well.

We could just not support these edge cases, but between:

1. exposing run_id + testRunID to cluster
2. having to change all the grafana dashboards + not supporting these two edge cases

The former seems like the less bad solution to me. So I opted to go with that, but I'm open to other's opinions though. For what it's worth, instead of directly passing run_id and testRunID as fields in cluster, I passed them via a grafanaTags field, to make it more generic + obvious this isn't something to be used outside of that context.

[srosenberg]

Nice! Do you have a screenshot of how this looks for the whole (test) run?

[DarrylWong]

Here's what follower-reads/mixed-version/single-region looks like.

Not sure how soon the data gets GC, but the grafana link in case you see this before it does.

[srosenberg]

Not sure how soon the data gets GC, but the grafana link in case you see this before it does.

Thanks. By default annotations are kept forever [1], and we're using the default. I'll add the following settings to make sure we don't forget to GC,

[annotations.dashboard]
max_age = 3M

[annotations.api]
max_age = 3M

[1] https://github.com/grafana/grafana/blob/608b40b2a8860c63f37fe5ac542b2a52b7109317/conf/sample.ini#L1241

[srosenberg]

Nice work! I've only gone through the code and left a few minor comments. I'll play around with the annotations to get a better idea of the "look & feel".

[srosenberg]

Annotations look good to me! If need be, we can easily tweak things in subsequent PR(s). Let's give @renatolabs a chance to review the changes since it touches the mixedversion runner. Otherwise, 🚢

[renatolabs]

Great work!

All my comments are ignorable nits.

[DarrylWong]

😞 Unfortunately adding a cluster.AddGrafanaAnnotation() call to runSingleStep breaks the Test_runSingleStep unit test since the unit test doesn't have a cluster mocked.

One easy solution would be to just wrap it in a if cluster != nil, but seems wrong to change it because it doesn't work with a test. On the other hand, mocking an entire cluster interface just to fix this one thing seems overkill? Would love to hear others thoughts, maybe I'm missing a better solution.

[renatolabs]

mocking an entire cluster interface just to fix this one thing seems overkill?

Yes, definitely an overkill. We've had to fix similar problems in the past; e.g., see how we mock the architecture for tests:

cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go

Lines 624 to 630 in f2985c3

	func (t *Test) clusterArch() vm.CPUArch {
	if t._arch != nil {
	return *t._arch // test-only
	}
	return t.cluster.Architecture()
	}

I kind of prefer having this type of behaviour encapsulated in a function rather than having != nil checks spread over the core logic. Maybe in this case we could have a _addAnnotation func() field and a function like:

func (r runner) addGrafanaAnnotation() {
    if r._addAnnotation != nil {
        r._addAnnotation(annotation) // test-only
        return
    }

    r.cluster.AddGrafanaAnnotation(annotation)
}

Needless to say, the above is pseudo-code to illustrate the idea. Does that make sense? You could then even inject your own annotation function in the test to validate the annotation we're sending.

[DarrylWong]

Yes that makes perfect sense! Thank you!

[DarrylWong]

CI is finally happy and confirmed it works on TC with the env variables.

TFTRs!

bors r=renatolabs, srosenberg

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)
• macos_amd64_build
• macos_arm64_build
• windows_build