changefeedccl: add roachtesting for network metrics

[asg0451]

Add a roachtest helper that checks that the new
cdc network metrics are being emitted. For now it
simply checks that they are > 0.

Part of: #130097

Release note: None

[cockroach-teamcity]

This change is 

[rharding6373]

Nice verifier! I have only one concern.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asg0451, @DarrylWong, and @renatolabs)

---

pkg/cmd/roachtest/tests/cdc.go line 1659 at r1 (raw file):

				steadyLatency:      5 * time.Minute,
			})
			ct.runMetricsVerifier(verifyChangefeedBytesInOutMetricsPresent("changefeed_network_bytes_in", "changefeed_network_bytes_out"))

What do you think about moving this after ct.waitForWorkload(). Although the changefeed has a job id at this point, it's not guaranteed that it has been scheduled, planned, and is processing events even with the 5s grace period as part of runMetricsVerifier. It will likely be less flaky if called at the end of the test, but that depends on if we're reading the metrics live or if we can get historical metrics.

Alternatively, could use testutils.SucceedsSoon to provide a longer grace period to runMetricsVerifier

---

pkg/cmd/roachtest/tests/cdc.go line 3712 at r1 (raw file):

}

func verifyChangefeedBytesInOutMetricsPresent(

nit: this function that checks metrics seems agnostic to bytes in/out. Might be better named something like verifyMetricNonZero.

[asg0451]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @renatolabs, and @rharding6373)

---

pkg/cmd/roachtest/tests/cdc.go line 1659 at r1 (raw file):

Previously, rharding6373 (Rachael Harding) wrote…

What do you think about moving this after ct.waitForWorkload(). Although the changefeed has a job id at this point, it's not guaranteed that it has been scheduled, planned, and is processing events even with the 5s grace period as part of runMetricsVerifier. It will likely be less flaky if called at the end of the test, but that depends on if we're reading the metrics live or if we can get historical metrics.

Alternatively, could use testutils.SucceedsSoon to provide a longer grace period to runMetricsVerifier

Fixed. The way i had it before was more convenient for testing it but this should work too and as you say be less flaky.

[rharding6373]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DarrylWong and @renatolabs)

[asg0451]

bors r=rharding6373

[craig]

Build succeeded:

• linux_amd64_build
• linux_amd64_fips_build
• local_roachtest_fips
• lint
• windows_build
• examples_orms
• linux_arm64_build
• check_generated_code
• macos_arm64_build
• docker_image_amd64
• unit_tests
• macos_amd64_build
• acceptance
• local_roachtest