kvstreamer: add some metrics

[yuzefovich]

This commit adds the following metrics:

• kv.streamer.operators.active tracks how many active streamer operators are out there currently
• kv.streamer.batches.count tracks total number of BatchRequests sent by all streamer instances
• kv.streamer.batches.in_progress tracks how many BatchRequests are in flight currently, across all streamer instances
• kv.streamer.batches.throttled tracks how many BatchRequests are being currently throttled (queued) due to reaching the streamer concurrency limit, across all streamer instances.

It also includes a couple of recently added dist sender metrics into roachprod opentelemetry registry.

Fixes: #131126.

Release note: None

[cockroach-teamcity]

This change is 

[yuzefovich]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @michae2)

---

pkg/kv/kvclient/kvstreamer/metrics.go line 18 at r1 (raw file):

	}
	metaStreamerRequestsInProgress = metric.Metadata{
		Name:        "kv.streamer.requests_in_progress",

I wonder whether we should use kv.streamers* prefix for the metrics. Unlike the dist sender (which is server-wide singleton), we create a separate streamer instance for every use, and e.g. the concurrency limit is applied to each instance separately, so perhaps plural in the metrics name would be more descriptive? Thoughts?

[michae2]

Nice!

Should we also add these to pkg/roachprod/opentelemetry/cockroachdb_metrics.go?

Reviewed 10 of 12 files at r1, 3 of 3 files at r2, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @yuzefovich)

---

pkg/kv/kvclient/kvstreamer/metrics.go line 18 at r1 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

I wonder whether we should use kv.streamers* prefix for the metrics. Unlike the dist sender (which is server-wide singleton), we create a separate streamer instance for every use, and e.g. the concurrency limit is applied to each instance separately, so perhaps plural in the metrics name would be more descriptive? Thoughts?

I think there's also a trend of naming stats after the "module", so from that perspective a singular kv.streamer prefix seems fine. I'm ok either way.

---

pkg/kv/kvclient/kvstreamer/metrics.go line 12 at r2 (raw file):

var (
	metaStreamerCount = metric.Metadata{
		Name:        "kv.streamer.operators.count",

bikeshedding: it's a little confusing to suffix a gauge metric with .count, maybe kv.streamer.operators.active would be clearer?

[yuzefovich]

Oh, TIL about the opentelemetry registry - added metrics from this PR as well as a couple we added earlier for the dist sender.

TFTR!

bors r+

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @DarrylWong, @herkolategan, and @michae2)

---

pkg/kv/kvclient/kvstreamer/metrics.go line 18 at r1 (raw file):

Previously, michae2 (Michael Erickson) wrote…

I think there's also a trend of naming stats after the "module", so from that perspective a singular kv.streamer prefix seems fine. I'm ok either way.

Ok, let's keep it singular then.

---

pkg/kv/kvclient/kvstreamer/metrics.go line 12 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

bikeshedding: it's a little confusing to suffix a gauge metric with .count, maybe kv.streamer.operators.active would be clearer?

Yeah, I think it's clearer, thanks.

[craig]

Build succeeded:

• windows_build
• linux_amd64_build
• lint
• local_roachtest_fips
• acceptance
• linux_amd64_fips_build
• linux_arm64_build
• macos_amd64_build
• examples_orms
• local_roachtest
• macos_arm64_build
• check_generated_code
• docker_image_amd64
• unit_tests