Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

admission: Provide better description on Overload metrics page #120853

Closed
Tracked by #121574
ajstorm opened this issue Mar 21, 2024 · 0 comments · Fixed by #123522
Closed
Tracked by #121574

admission: Provide better description on Overload metrics page #120853

ajstorm opened this issue Mar 21, 2024 · 0 comments · Fixed by #123522
Assignees
@aadityasondhi
Labels
A-admission-control A-cluster-observability Related to cluster observability C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-testcluster Issues found or occurred on a test cluster, i.e. a long-running internal cluster P-2 Issues/test failures with a fix SLA of 3 months T-admission-control Admission Control

Comments

@ajstorm
Copy link
Collaborator

ajstorm commented Mar 21, 2024 •
edited by exalate-issue-sync bot
Loading

The Overload metrics page of DB console has relatively sparse descriptions for the graphs, especially as you get further down the page and the graphs are generally more opaque. We should provide more robust descriptions for the graphs on this page. Examples:

image

image

Jira issue: CRDB-36902

Epic CRDB-36319
@ajstorm ajstorm added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-admission-control A-cluster-observability Related to cluster observability O-testcluster Issues found or occurred on a test cluster, i.e. a long-running internal cluster T-admission-control Admission Control P-2 Issues/test failures with a fix SLA of 3 months labels Mar 21, 2024
@aadityasondhi aadityasondhi self-assigned this Mar 26, 2024
@aadityasondhi aadityasondhi mentioned this issue Apr 2, 2024
Closed
4 tasks
@aadityasondhi aadityasondhi mentioned this issue May 2, 2024
Merged
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 2, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
0181397 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 7, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
5faf8b3 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 8, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
928550e 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 17, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
c69de76 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 17, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
a7e5919 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
aadityasondhi added a commit to aadityasondhi/cockroach that referenced this issue May 17, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
c006157 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes cockroachdb#120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
craig bot pushed a commit that referenced this issue May 21, 2024
@aadityasondhi @rickystewart
Merge #123522 #124493
02ca30c 
123522: dbconsole: overload page improvements r=sumeerbhola a=aadityasondhi

This PR contains a series of improvements to the overload page of the DB console as part of #121574. It is separated into multiple commits for ease of review.

____

dbconsole: remove non useful charts on the overload page

In investigations, we have found that the following charts are not
useful and frequently cause confusion:
- Admission work rate
- Admission Delay rate
- Requests Waiting For Flow Tokens

Informs #121572

Release note (ui change): This patch removes "Admission Delay Rate",
"Admission Work Rate", and "Requests Waiting For Flow Tokens". These
charts often cause confusion and are not useful for general overload
investigations.

___

dbconsole: reorder overload page metrics for better readability

This patch reorders the existing metrics in a more usable order:
1. Metrics to help determine which resource is constrained (IO, CPU)
2. Metrics to narrow down which AC queues are seeing requests waiting
3. More advanced metrics about the system health (goroutine scheduler,
   L0 sublevels, etc.)

Informs #121572.

Release note (ui change): Reordering of metrics on the overload page to
help categorizing them better. They are roughly in the following order:
1. Metrics to help determine which resource is constrained (IO, CPU)
2. Metrics to narrow down which AC queues are seeing requests waiting
3. More advanced metrics about the system health (goroutine scheduler,
   L0 sublevels, etc.)

___

dbconsole: include better names and descriptions for overload page
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes #120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.

___

dbconsole: additional higher granularity metrics for overload

This patch adds additional metrics to the overload page that allow for
more granular look at the system:
- cr.store.storage.l0-sublevels
- cr.node.go.scheduler_latency-p99.9

Informs #121572.

Release note (ui change): Two additional metrics on the overload page
for better visibility into overloaded resources:
- cr.store.storage.l0-sublevels
- cr.node.go.scheduler_latency-p99.9

___

dbconsole: split Admission Queue graphs to avoid overcrowding

Informs #121572.

Release note (ui change): There are now 4 graphs for Admission Queue
Delay:
1. Foreground (regular) CPU work
2. Store (IO) work
3. Background (elastic) CPU work
4. Replication Admission Control, store overload on replicas

___

dbconsole: add elastic store metric to the overload page 

This patch uses the new sperated `elastic-stores` metrics for queing
delay from #123890.

Informs #121572.

Release note (ui change): The `Admission Queueing Delay – Store` chart
now separates elastic (background) work from the regular foreground
work.

___

dbconsole: add elastic io token exhausted duration to overload page 

This patch adds the metric `elastic_io_tokens_exhausted_duration.kv`
introduced in #124078.

Informs #121572.

Release note (ui change): The `Admission IO Tokens Exhausted` chart now
separates elastic and regular io work.

124493: packer: only try emulating via Docker on x86 r=rail a=rickystewart

Epic: none

Release note: None

Co-authored-by: Aaditya Sondhi <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
@craig craig bot closed this as completed in 01c989a May 21, 2024
blathers-crl bot pushed a commit that referenced this issue May 21, 2024
@aadityasondhi
dbconsole: include better names and descriptions for overload page
92f8aa8 
This patch improves the metric descriptions for the metrics on the
overload page.

Fixes #120853.

Release note (ui change): The overload page now includes descriptions for all
metrics.
@blathers-crl blathers-crl bot mentioned this issue May 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aadityasondhi aadityasondhi

Labels
A-admission-control A-cluster-observability Related to cluster observability C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-testcluster Issues found or occurred on a test cluster, i.e. a long-running internal cluster P-2 Issues/test failures with a fix SLA of 3 months T-admission-control Admission Control
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

dbconsole: overload page improvements aadityasondhi/cockroach
2 participants
@ajstorm @aadityasondhi