server: add node_id label to _status/vars output

[dhartunian]

Previously, the output of the prometheus metrics via _status/ vars did not include any node labels. This caused challenges for customers who want to monitor large clusters as it requires additional configuration on the scrape- side to ensure a node ID is added to the metrics. This can be challenging to deal with when nodes come and go in a cluster and the scrape configuration must change as well.

This change adds a node_id prometheus label to the metrics we output that matches the current node's ID. Since _status/vars is output from a single node there is only ever one single value that's appropriate here.

Secondary tenants will mark their metrics with either the nodeID of the shared- process system tenant, or the instanceID of the tenant process.

Resolves: #94763
Epic: None

Release note (ops change): Prometheus metrics available at the _status/vars path now contain a node_id label that identifies the node they were scraped from.

[cockroach-teamcity]

This change is 

[aadityasondhi]

I think you will have to update a bunch of tests with the new NodeID label (CI failures)

Reviewable status: complete! 1 of 0 LGTMs obtained

[dhartunian]

bors r=aadityasondhi

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[knz]

bors r=aadityasondhi single on

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[adityamaru]

@dhartunian the batch seems to be failing on TestMetricsRecorderLabels:

 recorder_test.go:240: recorder did not yield expected time series collection; diff:
         [[0].Source: "1" != "7" [0].Datapoints[0].Value: 1 != 7 [1].Source: "1-123" != "7-123" [2].Source: "1" != "7" [3].Source: "1-123" != "7-123"]

[dhartunian]

bors r-

[craig]

Canceled.

[aadityasondhi]

Sorry, I think my tsdb changes landed prior to this. The source fields now have the Tenant IDs in them. #98077

[dhartunian]

bors r=aadityasondhi

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}