roachtest: expose an API to label (cluster) VMs with name of executing test

[srosenberg]

For observability purposes, we need to map a test to the cluster where it was executed. Roachprod already supports custom VM labels across all cloud providers. Hence, it should be straightforward to expose an internal API, that's accessible to roachtest.
Note that a cluster may be reused, hence care should be taken to ensure the label is removed upon test completion.
Prometheus config. should be updated to scrape the new label.

In addition to making it easy to pull up the metrics for a given test run in Grafana, this would also support fine-grained billing attribution (by test name).

Informs: #98658

Jira issue: CRDB-28489

[blathers-crl]

cc @cockroachdb/test-eng

[tbg]

Came here for +1. I'd like to know what ran on teamcity-10706494-1687931234-31-n3cpu8 on 7/28 3pm and I can find out by going through roachtest nightlies but it's tedious. (I'm not asking anyone to do it for me, just giving an example of the workflow Stan no doubt had in mind when he filed this).

[renatolabs]

@smg260 #107965 should have closed this one, yes?