Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jobsprofiler: collect cluster-wide stacks for a given job #105440

Closed
adityamaru opened this issue Jun 23, 2023 · 1 comment
Closed

jobsprofiler: collect cluster-wide stacks for a given job #105440

adityamaru opened this issue Jun 23, 2023 · 1 comment
Assignees
@adityamaru
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-jobs

Comments

@adityamaru
Copy link
Contributor

adityamaru commented Jun 23, 2023 •
edited
Loading

The job details page in the DBConsole should have a button to collect/download cluster-wide stacks labeled with the jobID as the pprof label. These stacks can also be added to the job bundle.

Epic: CRDB-8964

Jira issue: CRDB-29048
@adityamaru adityamaru added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-jobs labels Jun 23, 2023
@blathers-crl blathers-crl bot added the A-jobs label Jun 23, 2023
@adityamaru adityamaru mentioned this issue Jun 30, 2023
Merged
adityamaru added a commit to adityamaru/cockroach that referenced this issue Jul 1, 2023
@adityamaru
pprofui: add support for collecting goroutines with labels
c618d44 
This change refactors some of the logic in the pprofui/.
Additionally, we add support for collecting cluster-wide
goroutine profiles.

These goroutine profiles can be collected with or without labels.
When collected without labels we request pprof profiles
with debug=0, which generates Profile protobufs on every
node and `pprof.Merge`s them before redirecting the client
to a flamegraph.

When collected with labels we request pprof profiles with
debug=1, which generates a legacy text format with comments
translating addresses to function names and line numbers,
so that a programmer can read the profile without tools.
These profiles cannot be `pprof.Merged` and so we manually
stitch them together on a per-node basis before downloading
them for the client as txt file.

This change also adds support for filtering the aforementioned
goroutines by pprof label. This will be used by the Jobs Profiler
to collect cluster-wide stacks relevant to the running job.

Informs: cockroachdb#105440
Release note: None
adityamaru added a commit to adityamaru/cockroach that referenced this issue Jul 5, 2023
@adityamaru
pprofui: add support for collecting goroutines with labels
b25619f 
This change refactors some of the logic in the pprofui/.
Additionally, we add support for collecting cluster-wide
goroutine profiles.

These goroutine profiles can be collected with or without labels.
When collected without labels we request pprof profiles
with debug=0, which generates Profile protobufs on every
node and `pprof.Merge`s them before redirecting the client
to a flamegraph.

When collected with labels we request pprof profiles with
debug=1, which generates a legacy text format with comments
translating addresses to function names and line numbers,
so that a programmer can read the profile without tools.
These profiles cannot be `pprof.Merged` and so we manually
stitch them together on a per-node basis before downloading
them for the client as txt file.

This change also adds support for filtering the aforementioned
goroutines by pprof label. This will be used by the Jobs Profiler
to collect cluster-wide stacks relevant to the running job.

Informs: cockroachdb#105440
Release note: None
adityamaru added a commit to adityamaru/cockroach that referenced this issue Jul 6, 2023
@adityamaru
pprofui: add support for collecting goroutines with labels
b38951a 
This change refactors some of the logic in the pprofui/.
Additionally, we add support for collecting cluster-wide
goroutine profiles.

These goroutine profiles can be collected with or without labels.
When collected without labels we request pprof profiles
with debug=0, which generates Profile protobufs on every
node and `pprof.Merge`s them before redirecting the client
to a flamegraph.

When collected with labels we request pprof profiles with
debug=1, which generates a legacy text format with comments
translating addresses to function names and line numbers,
so that a programmer can read the profile without tools.
These profiles cannot be `pprof.Merged` and so we manually
stitch them together on a per-node basis before downloading
them for the client as txt file.

This change also adds support for filtering the aforementioned
goroutines by pprof label. This will be used by the Jobs Profiler
to collect cluster-wide stacks relevant to the running job.

Informs: cockroachdb#105440
Release note: None
craig bot pushed a commit that referenced this issue Jul 7, 2023
@adityamaru
Merge #105916
5aeb360 
105916: pprofui: add support for collecting goroutines with labels r=tbg a=adityamaru

This change refactors some of the logic in the pprofui/. Additionally, we add support for collecting cluster-wide goroutine profiles.

These goroutine profiles can be collected with or without labels. When collected without labels we request pprof profiles with debug=0, which generates Profile protobufs on every node and `pprof.Merge`s them before redirecting the client to a flamegraph.

When collected with labels we request pprof profiles with debug=1, which generates a legacy text format with comments translating addresses to function names and line numbers, so that a programmer can read the profile without tools. These profiles cannot be `pprof.Merged` and so we manually stitch them together on a per-node basis before downloading them for the client as txt file.

This change also adds support for filtering the aforementioned goroutines by pprof label. This will be used by the Jobs Profiler to collect cluster-wide stacks relevant to the running job.

Informs: #105440
Release note: None

Co-authored-by: adityamaru <[email protected]>
@adityamaru adityamaru mentioned this issue Jul 11, 2023
Merged
@adityamaru
Copy link
Contributor Author

This is done in #105916 and #106654

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adityamaru adityamaru

Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-jobs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@adityamaru