sql,server: add endpoint to list a job's execution details #106629
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adityamaru
requested review from
a team and
ericharmeling
and removed request for
a team
|
This change is |
dt
reviewed
Jul 12, 2023
dt
reviewed
Jul 12, 2023
dt
approved these changes
Jul 12, 2023
knz
reviewed
Jul 12, 2023
There was a problem hiding this comment.
Reviewed 2 of 6 files at r1.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @adityamaru and @ericharmeling)
pkg/sql/jobs_profiler_bundle.go line 172 at r1 (raw file):
// Look for the final chunk of each file to find the unique file name. if idx := strings.Index(infoKey, finalChunkSuffix); idx != -1 { files = append(files, strings.TrimPrefix(infoKey[:idx], profilerconstants.ExecutionDetailsChunkKeyPrefix))
I'm worried by the cumulative size of this data. Also this code is not using memory accounting.
Would it make sense to perhaps use a streaming API instead?
I don't really anticipate this being an issue in near term. We're probably going to add support for ~10 things (stretch goal) we collect as part of a job's execution details in this release. That coupled with work being done to delete older, ephemeral rows in the job_info table #102640 should mean that we're shipping around a slice of bounded size strings. There is a path to add a |
|
yeah, I don't see the list of file names being what will OOM us; e.g. we're compressing 1mb chunks when we read and write files, and it seems unlikely the list of file names will get anywhere close to 1mb any time soon, let alone coming us. I think we can defer worrying about this for some time (and maybe forever), until the list of files is much bigger, and then as @adityamaru says, there is a path to adding limit/offset to this API for pagination when we get there. Switching to a more complex stream API now feels like premature optimization. |
In cockroachdb#105384 we added infrastructure to request and store execution details for a job. This currently only includes the DistSQL diagram generated during a job execution. Going forward this will include several files such as traces, goroutines, profiles etc. This change introduces an endpoint that allows listing all such files that are available for consumption. This list will be displayed on the job details page allowing the user to download any subset of the files collected during job execution. Informs: cockroachdb#105076 Release note: None
adityamaru
force-pushed
the
add-list-endpoint
branch
from
a8b2636 to
62038ac
Compare
|
flake is #106736 bors r=dt |
|
Build succeeded: |
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this pull request
Jul 17, 2023
In cockroachdb#105384 and cockroachdb#106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved obersvability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A future change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. Informs: cockroachdb#105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this pull request
Jul 17, 2023
In cockroachdb#105384 and cockroachdb#106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved observability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A follow-up change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. This change also renames the `Profiler` tab to `Advanced Debugging` as the users of this tab are going to be internal CRDB support and engineering for the time being. Informs: cockroachdb#105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this pull request
Jul 18, 2023
In cockroachdb#105384 and cockroachdb#106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved observability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A follow-up change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. This change also renames the `Profiler` tab to `Advanced Debugging` as the users of this tab are going to be internal CRDB support and engineering for the time being. Informs: cockroachdb#105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this pull request
Jul 19, 2023
In cockroachdb#105384 and cockroachdb#106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved observability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A follow-up change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. This change also renames the `Profiler` tab to `Advanced Debugging` as the users of this tab are going to be internal CRDB support and engineering for the time being. Informs: cockroachdb#105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this pull request
Jul 24, 2023
In cockroachdb#105384 and cockroachdb#106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved observability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A follow-up change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. This change also renames the `Profiler` tab to `Advanced Debugging` as the users of this tab are going to be internal CRDB support and engineering for the time being. Informs: cockroachdb#105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details
craig bot
pushed a commit
that referenced
this pull request
Jul 24, 2023
106879: jobs: add table to display execution details r=maryliag a=adityamaru In #105384 and #106629 we added support to collect and list files that had been collected as part of a job's execution details. These files are meant to provide improved obersvability into the state of a job. This change is the first of a few that exposes these endpoints on the DBConsole job details page. This change only adds support for listing files that have been requested as part of a job's execution details. A future change will add support to request these files, sort them and download them from the job details page. This page is not available on the Cloud Console as it is meant for advanced debugging. Informs: #105076 Release note (ui change): add table in the Profiler job details page that lists all the available files describing a job's execution details <img width="1505" alt="Screenshot 2023-07-18 at 2 26 50 PM" src="https://github.com/cockroachdb/cockroach/assets/13837382/aebe18a6-9c25-4c9a-ad7c-a94e2e4c97ff"> <img width="1510" alt="Screenshot 2023-07-18 at 2 27 03 PM" src="https://github.com/cockroachdb/cockroach/assets/13837382/da9b3a21-8dc6-47ca-ac02-24d8bb7d09e7"> 107236: sql: use txn.NewBatch instead of &kv.Batch{} r=fqazi a=rafiss This will make these requests properly passes along the admission control headers. informs #79212 Epic: None Release note: None 107447: sql: fix CREATE MATERIALIZED VIEW AS schema change job description r=fqazi a=ecwall Fixes #107445 This changes the CREATE MATERIALIZED VIEW AS schema change job description SQL syntax. For example ``` CREATE VIEW "v" AS "SELECT t.id FROM movr.public.t"; ``` becomes ``` CREATE MATERIALIZED VIEW defaultdb.public.v AS SELECT t.id FROM defaultdb.public.t WITH DATA; ``` Release note (bug fix): Fix CREATE MATERIALIZED VIEW AS schema change job description SQL syntax. Co-authored-by: adityamaru <[email protected]> Co-authored-by: Rafi Shamim <[email protected]> Co-authored-by: Evan Wall <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #105384 we added infrastructure to request and store execution details for a job. This currently only includes the DistSQL diagram generated during a job execution. Going forward this will include several files such as traces, goroutines, profiles etc.
This change introduces an endpoint that allows listing all such files that are available for consumption. This list will be displayed on the job details page allowing the user to download any subset of the files collected during job execution.
Informs: #105076
Release note: None