PR #109835 - changefeedccl: add changefeed.lagging_ranges metric

[cockroach-teamcity]

Exalate commented:

Related PR: cockroachdb/cockroach#109835
Commit: cockroachdb/cockroach@b5ec7df
Epic: none

---

Release note (ops change): A new metric changefeed.lagging_ranges is added to show the number of
ranges which are behind in changefeeds. This metric can be used with the metrics_label changefeed
option. A changefeed option lagging_ranges_threshold is added which is the amount of
time a range needs to be behind to be considered lagging. By default this is 3 minutes. There is also
a new option lagging_ranges_polling_interval which controls how often the lagging ranges
calculation is done. This setting defaults to polling every 1 minute.

Note that polling adds latency to the metric being updated. For example, if a range falls behind
by 3 minutes, the metric may not update until an additional minute afterwards.

Also note that ranges undergoing an initial scan for longer than the threshold are considered to be
lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric
for each range in the table. However, as ranges complete the initial scan, the number of ranges will
decrease.

Epic: None

Jira Issue: DOC-8745