jobs/cdc: add metrics for paused jobs #89752

jayshrivastava · 2022-10-11T15:14:38Z

This change adds new metrics to count paused jobs for every job type. For
example, the metric for paused changefeed jobs is
jobs.changefeed.currently_paused. These metrics are counted at an
interval defined by the cluster setting jobs.metrics.interval.poll.

This is implemented by a job which periodically queries system.jobs
to count the number of paused jobs. This job is of the newly added type
jobspb.TypePollJobsStats. When a node starts it's job registry, it will
create an adoptable stats polling job if it does not exist already using a
transaction.

This change adds a test which pauses and resumes changefeeds while asserting
the value of the jobs.changefeed.currently_paused metric. It also adds a
logictest to ensure one instance of the stats polling job is created in a
cluster.

Resolves: #85467

Release note (general change): This change adds new metrics to count
paused jobs for every job type. For example, the metric for paused
changefeed jobs is jobs.changefeed.currently_paused. These metrics
are updated at an interval defined by the cluster setting
jobs.metrics.interval.poll, which is defauled to 10 seconds.

Epic: None

jayshrivastava · 2022-10-11T15:14:43Z

edit (v2):

cockroach-teamcity · 2022-10-11T15:14:48Z

This change is

pkg/ccl/changefeedccl/metrics.go

pkg/ccl/changefeedccl/changefeed_stmt.go

jayshrivastava · 2022-10-13T20:48:56Z

Seems that we already have metrics for this #85467 (comment). Closing for now.

miretskiy · 2022-10-13T21:22:28Z

I don't think this is correct: idleness is an entirely different concept related to the number of jobs that are currently running, but that are idle -- so that they can be shutdown by serverless. This has nothing to do with this pr.

…

On Thu, Oct 13, 2022 at 4:53 PM Jayant Shrivastava ***@***.***> wrote: Seems that we already have metrics for this #85467 (comment) <#85467 (comment)>. Closing for now. — Reply to this email directly, view it on GitHub <#89752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANA4FVCO7YJ2OU5QC5DV5ZTWDBZD7ANCNFSM6AAAAAARCMUUKI> . You are receiving this because your review was requested.Message ID: ***@***.***>

pkg/jobs/adopt.go

jayshrivastava · 2022-10-19T14:41:09Z

Leaving a note for when I get back to this. https://github.com/jayshrivastava/cockroach/tree/rowfetcher-2
Need to work on

adding a virtual index on status to the internal jobs table so we don't have to scan the whole thing
scheduling this metric update less often and also having one node do it

jayshrivastava

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @miretskiy, @samiskin, and @shermanCRL)

pkg/jobs/registry.go line 934 at r5 (raw file):

Previously, miretskiy (Yevgeniy Miretskiy) wrote…

you probably want to set noncancellable bit?

Done.

pkg/jobs/registry.go line 972 at r5 (raw file):

Previously, miretskiy (Yevgeniy Miretskiy) wrote…

right; but you're not declaring; you're creating it.
Do declare:
var metricUpdate map[jobspb.Type]int

Done.

pkg/jobs/testing_knobs.go line 93 at r5 (raw file):

Previously, miretskiy (Yevgeniy Miretskiy) wrote…

not sure we need a pointer -- i guess it's consistent, so ... fine...

Done.

miretskiy

Reviewed 9 of 28 files at r8, 4 of 6 files at r9, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @jayshrivastava and @samiskin)

pkg/jobs/registry.go line 1148 at r9 (raw file):

				err = ctx.Err()
				return
			case <-time.After(PollJobsMetricsInterval.Get(&r.settings.SV)):

you probably want to create timer for this.

pkg/upgrade/upgrades/create_jobs_metrics_polling_job.go line 44 at r9 (raw file):

		}

		// If there isn't a row for the key visualizer job, create the job.

coment needs updating.

I suspect this code is repeated with key visualizer logic? Consider adding a helper (createBootstrapJob or some such)

jayshrivastava · 2023-02-07T16:50:05Z

bors r=miretskiy TYFR!

craig · 2023-02-07T17:43:05Z

Build failed (retrying...):

Bazel Essential CI (Cockroach)

craig · 2023-02-07T18:12:36Z

Build failed (retrying...):

Bazel Essential CI (Cockroach)

jayshrivastava · 2023-02-07T19:02:36Z

bors r-

craig · 2023-02-07T19:02:40Z

Canceled.

This change adds new metrics to count paused jobs for every job type. For example, the metric for paused changefeed jobs is `jobs.changefeed.currently_paused`. These metrics are counted at an interval defined by the cluster setting `jobs.metrics.interval.poll`. This is implemented by a job which periodically queries `system.jobs` to count the number of paused jobs. This job is of the newly added type `jobspb.TypePollJobsStats`. When a node starts it's job registry, it will create an adoptable stats polling job if it does not exist already using a transaction. This change adds a test which pauses and resumes changefeeds while asserting the value of the `jobs.changefeed.currently_paused` metric. It also adds a logictest to ensure one instance of the stats polling job is created in a cluster. Resolves: cockroachdb#85467 Release note (general change): This change adds new metrics to count paused jobs for every job type. For example, the metric for paused changefeed jobs is `jobs.changefeed.currently_paused`. These metrics are updated at an interval defined by the cluster setting `jobs.metrics.interval.poll`, which is defauled to 10 seconds. Epic: None

jayshrivastava · 2023-02-08T16:35:08Z

bors r=miretskiy

craig · 2023-02-08T18:15:12Z

Build succeeded:

Bazel Essential CI (Cockroach)

Prior PR cockroachdb#89752 added a metrics poller job which produces per job type stats on the number of paused jobs. This PR extends metrics poller to also collect stats related to protected timestamps created by jobs. Namely, two new metrics, per job type are added: * `jobs.<job type>.protected_record_count` -- keeps track of the number of protected timestamp records help by the jobs. * `jobs.<job type>.protected_age_sec` -- keeps track of the age of the oldest protected timestamp held by those jobs. The metrics improve observability into protected timestamp system, and allow operators to alert when protected timestamp records are too old since that prevents garbage collection from occuring (and if GC is not performed for too long, the cluster performance would degrade). Follow on work will also make this functionality available for schedules. Epic: CRDB-21953 Fixes cockroachdb#78354 Release note (enterprise change): Jobs that utilize protected timestamp system (BACKUP, CHANGEFEED, IMPORT, etc) now produce metrics that can be monitored to detect cases when job leaves stale protected timestamp, preventing garbage collection from occuring.

jayshrivastava requested a review from samiskin October 11, 2022 15:15

jayshrivastava marked this pull request as ready for review October 11, 2022 15:15

jayshrivastava requested a review from a team as a code owner October 11, 2022 15:15

jayshrivastava requested review from shermanCRL and removed request for a team October 11, 2022 15:15

miretskiy suggested changes Oct 11, 2022

View reviewed changes

pkg/ccl/changefeedccl/metrics.go Outdated Show resolved Hide resolved

pkg/ccl/changefeedccl/changefeed_stmt.go Outdated Show resolved Hide resolved

jayshrivastava closed this Oct 13, 2022

miretskiy reopened this Oct 13, 2022

jayshrivastava force-pushed the paused-metrics branch from 297bc36 to a557268 Compare October 14, 2022 17:54

jayshrivastava requested a review from a team as a code owner October 14, 2022 17:54

jayshrivastava force-pushed the paused-metrics branch 3 times, most recently from 211a9cb to 8cd140d Compare October 14, 2022 18:11

jayshrivastava mentioned this pull request Oct 14, 2022

jobs: add metric for number of paused jobs #85467

Closed

jayshrivastava force-pushed the paused-metrics branch 4 times, most recently from a35029b to 1706650 Compare October 17, 2022 15:15

miretskiy suggested changes Oct 17, 2022

View reviewed changes

pkg/jobs/adopt.go Outdated Show resolved Hide resolved

jayshrivastava force-pushed the paused-metrics branch from 1706650 to eb534e2 Compare October 17, 2022 15:23

jayshrivastava force-pushed the paused-metrics branch from eb534e2 to 3082ac7 Compare October 31, 2022 20:38

jayshrivastava requested a review from a team October 31, 2022 20:38

jayshrivastava requested review from a team as code owners October 31, 2022 20:38

jayshrivastava changed the title ~~changefeedccl: add metrics for paused changefeed jobs~~ jobs/cdc: add metrics for paused jobs Oct 31, 2022

jayshrivastava force-pushed the paused-metrics branch 3 times, most recently from 9411852 to fb89d23 Compare February 6, 2023 21:13

jayshrivastava commented Feb 6, 2023

View reviewed changes

jayshrivastava force-pushed the paused-metrics branch 2 times, most recently from 369144c to ef93a0f Compare February 6, 2023 21:55

shermanCRL removed their request for review February 7, 2023 15:03

miretskiy approved these changes Feb 7, 2023

View reviewed changes

jayshrivastava force-pushed the paused-metrics branch from ef93a0f to 0360c2c Compare February 7, 2023 15:50

jayshrivastava force-pushed the paused-metrics branch from 0360c2c to 87fcb73 Compare February 7, 2023 20:00

jayshrivastava force-pushed the paused-metrics branch from 87fcb73 to 690da3e Compare February 8, 2023 15:33

craig bot merged commit 844a370 into cockroachdb:master Feb 8, 2023

jayshrivastava deleted the paused-metrics branch February 8, 2023 18:15

cockroach-teamcity mentioned this pull request Feb 9, 2023

PR #89752 - jobs: add metrics for paused jobs cockroachdb/docs#16227

Closed

jayshrivastava mentioned this pull request Feb 13, 2023

cdc: max_behind_nanos does not update when jobs are stuck during initial scans #97043

Open

jayshrivastava mentioned this pull request Feb 15, 2023

changefeedccl: add metric for paused changefeeds #86789

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jobs/cdc: add metrics for paused jobs #89752

jobs/cdc: add metrics for paused jobs #89752

jayshrivastava commented Oct 11, 2022 •

edited

Loading

jayshrivastava commented Oct 11, 2022 •

edited

Loading

cockroach-teamcity commented Oct 11, 2022

jayshrivastava commented Oct 13, 2022

miretskiy commented Oct 13, 2022 via email

jayshrivastava commented Oct 19, 2022

jayshrivastava left a comment

miretskiy left a comment

jayshrivastava commented Feb 7, 2023

craig bot commented Feb 7, 2023

craig bot commented Feb 7, 2023

jayshrivastava commented Feb 7, 2023

craig bot commented Feb 7, 2023

jayshrivastava commented Feb 8, 2023

craig bot commented Feb 8, 2023

jobs/cdc: add metrics for paused jobs #89752

jobs/cdc: add metrics for paused jobs #89752

Conversation

jayshrivastava commented Oct 11, 2022 • edited Loading

jayshrivastava commented Oct 11, 2022 • edited Loading

cockroach-teamcity commented Oct 11, 2022

jayshrivastava commented Oct 13, 2022

miretskiy commented Oct 13, 2022 via email

jayshrivastava commented Oct 19, 2022

jayshrivastava left a comment

Choose a reason for hiding this comment

miretskiy left a comment

Choose a reason for hiding this comment

jayshrivastava commented Feb 7, 2023

craig bot commented Feb 7, 2023

craig bot commented Feb 7, 2023

jayshrivastava commented Feb 7, 2023

craig bot commented Feb 7, 2023

jayshrivastava commented Feb 8, 2023

craig bot commented Feb 8, 2023

jayshrivastava commented Oct 11, 2022 •

edited

Loading

jayshrivastava commented Oct 11, 2022 •

edited

Loading