jobs: publish max protected timestamp as a metric #78354
Assignees
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-jobs
Comments
amruss
added
the
C-enhancement
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Feb 15, 2023
Prior PR cockroachdb#89752 added a metrics poller job which produces per job type stats on the number of paused jobs. This PR extends metrics poller to also collect stats related to protected timestamps created by jobs. Namely, two new metrics, per job type are added: * `jobs.<job type>.protected_record_count` -- keeps track of the number of protected timestamp records help by the jobs. * `jobs.<job type>.protected_age_sec` -- keeps track of the age of the oldest protected timestamp held by those jobs. The metrics improve observability into protected timestamp system, and allow operators to alert when protected timestamp records are too old since that prevents garbage collection from occuring (and if GC is not performed for too long, the cluster performance would degrade). Follow on work will also make this functionality available for schedules. Epic: CRDB-21953 Fixes cockroachdb#78354 Release note (enterprise change): Jobs that utilize protected timestamp system (BACKUP, CHANGEFEED, IMPORT, etc) now produce metrics that can be monitored to detect cases when job leaves stale protected timestamp, preventing garbage collection from occuring.
miretskiy
pushed a commit
to miretskiy/cockroach
that referenced
this issue
Feb 16, 2023
Prior PR cockroachdb#89752 added a metrics poller job which produces per job type stats on the number of paused jobs. This PR extends metrics poller to also collect stats related to protected timestamps created by jobs. Namely, two new metrics, per job type are added: * `jobs.<job type>.protected_record_count` -- keeps track of the number of protected timestamp records help by the jobs. * `jobs.<job type>.protected_age_sec` -- keeps track of the age of the oldest protected timestamp held by those jobs. The metrics improve observability into protected timestamp system, and allow operators to alert when protected timestamp records are too old since that prevents garbage collection from occuring (and if GC is not performed for too long, the cluster performance would degrade). Follow on work will also make this functionality available for schedules. Epic: CRDB-21953 Fixes cockroachdb#78354 Release note (enterprise change): Jobs that utilize protected timestamp system (BACKUP, CHANGEFEED, IMPORT, etc) now produce metrics that can be monitored to detect cases when job leaves stale protected timestamp, preventing garbage collection from occuring.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Users should be able to alert on how far behind we're protecting jobs, to protect against the scenario where a job is causing an unacceptable amount of garabage.
Jira issue: CRDB-14088
Epic CRDB-21953
The text was updated successfully, but these errors were encountered: