Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rangefeed: add metrics to rangefeed scheduler #110420

Closed
aliher1911 opened this issue Sep 12, 2023 · 1 comment · Fixed by #110458
Closed

rangefeed: add metrics to rangefeed scheduler #110420

aliher1911 opened this issue Sep 12, 2023 · 1 comment · Fixed by #110458
Assignees
@aliher1911
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@aliher1911
Copy link
Contributor

aliher1911 commented Sep 12, 2023 •
edited by exalate-issue-sync bot
Loading

Current scheduler doesn't expose any metrics. This could be limiting as we don't have a visibility on how long rangefeed processors could wait in the queue before their work is scheduled for execution.

We need to at least have a wait time histogram and a pending queue size.

This visibility is critical to measuring how feeds could affect eachother and how timely system events are delivered over rangefeeds.

Jira issue: CRDB-31419

Epic CRDB-26372
@aliher1911 aliher1911 added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels Sep 12, 2023
@aliher1911 aliher1911 self-assigned this Sep 12, 2023
@blathers-crl
Copy link

blathers-crl bot commented Sep 12, 2023

cc @cockroachdb/replication

@aliher1911 aliher1911 mentioned this issue Sep 12, 2023
Merged
aliher1911 added a commit to aliher1911/cockroach that referenced this issue Sep 20, 2023
@aliher1911
rangefeed: add rangefeed scheduler metrics
b3f1f10 
This commit adds two rangefeed scheduler metrics to assist performance
debugging.
Scheduler latency histogram `kv.rangefeed.scheduler.latency` which is
showing how long rangefeed processor events wait in queue before they
are schedued for processing.
Scheduler queue size `kv.rangefeed.scheduler.queue_size` showing how
many ranges pending event processing.

Epic: cockroachdb#110420

Release note: None
aliher1911 added a commit to aliher1911/cockroach that referenced this issue Sep 20, 2023
@aliher1911
rangefeed: add rangefeed scheduler metrics
569b3a3 
This commit adds two rangefeed scheduler metrics to assist performance
debugging.
Scheduler latency histogram `kv.rangefeed.scheduler.latency` which is
showing how long rangefeed processor events wait in queue before they
are schedued for processing.
Scheduler queue size `kv.rangefeed.scheduler.queue_size` showing how
many ranges pending event processing.

Epic: cockroachdb#110420

Release note: None
aliher1911 added a commit to aliher1911/cockroach that referenced this issue Sep 22, 2023
@aliher1911
rangefeed: add rangefeed scheduler metrics
140b248 
This commit adds two rangefeed scheduler metrics to assist performance
debugging.
Scheduler latency histogram `kv.rangefeed.scheduler.latency` which is
showing how long rangefeed processor events wait in queue before they
are schedued for processing.
Scheduler queue size `kv.rangefeed.scheduler.queue_size` showing how
many ranges pending event processing.

Epic: cockroachdb#110420

Release note: None
aliher1911 added a commit to aliher1911/cockroach that referenced this issue Sep 22, 2023
@aliher1911
rangefeed: add rangefeed scheduler metrics
9a4096f 
This commit adds two rangefeed scheduler metrics to assist performance
debugging.
Latency histogram which is showing how long events spent waiting in queue
before they are scheduled for processing:
`kv.rangefeed.scheduler.normal-priority.latency`
`kv.rangefeed.scheduler.system-priority.latency`
Queue size which is showing how many ranges are pending event to be processed:
`kv.rangefeed.scheduler.normal-priority.latency`
`kv.rangefeed.scheduler.system-priority.latency`

Epic: cockroachdb#110420

Release note: None
aliher1911 added a commit to aliher1911/cockroach that referenced this issue Sep 22, 2023
@aliher1911
rangefeed: add rangefeed scheduler metrics
ab00ae9 
This commit adds two rangefeed scheduler metrics to assist performance
debugging.
Latency histogram which is showing how long events spent waiting in queue
before they are scheduled for processing:
`kv.rangefeed.scheduler.normal.latency`
`kv.rangefeed.scheduler.system.latency`
Queue size which is showing how many ranges are pending event to be processed:
`kv.rangefeed.scheduler.normal.queue_size`
`kv.rangefeed.scheduler.system.queue_size`

Epic: cockroachdb#110420

Release note: None
@craig craig bot closed this as completed in 235babd Sep 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliher1911 aliher1911

Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rangefeed: add rangefeed scheduler metrics aliher1911/cockroach
1 participant
@aliher1911