Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: add parent_job parameter to list_jobs #9190

Closed
tswast opened this issue Sep 9, 2019 · 1 comment · Fixed by #9225
Closed

BigQuery: add parent_job parameter to list_jobs #9190

tswast opened this issue Sep 9, 2019 · 1 comment · Fixed by #9225
Assignees
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Contributor

tswast commented Sep 9, 2019

Is your feature request related to a problem? Please describe.

With the scripting feature (alpha), a single query job can initiate several sub-jobs. These job resources are not embedded in the initial job. Instead, you must populate the parentJobId query parameter to the BigQuery jobs list API.

Describe the solution you'd like

I propose a new parent_job parameter to Client.list_jobs.

  • Accept a string (interpreted as a job ID).
  • Accept a *Job object. Use the job ID from the job object. Also, use the location from the job object.

Describe alternatives you've considered

We could make a list_children() method on the job classes, but we'd still need to update list_jobs to support it. Let's start just updating list_jobs for now.

Additional context

Example from the alpha docs.

BigQuery’s scripting feature enables you to send multiple statements to BigQuery in one request, to use variables, and to use control flow statements such as IF and WHILE. For example, you can declare a variable, assign a value to it, and then reference it in a third statement:

-- Declare a variable to hold names as an array.
DECLARE top_names ARRAY<STRING>;

-- Build an array of the top 100 names from the year 2017.
SET top_names = (
  SELECT ARRAY_AGG(name ORDER BY number DESC LIMIT 100)
  FROM `bigquery-public-data`.usa_names.usa_1910_current
  WHERE year = 2017
);

-- Which names appear as words in Shakespeare's plays?
SELECT
  name AS shakespeare_name
FROM UNNEST(top_names) AS name
WHERE name IN (
  SELECT word
  FROM `bigquery-public-data`.samples.shakespeare
);

For projects that have enrolled in the scripting alpha, BigQuery interprets any request with multiple statements as a script, unless the statements comprise some number of CREATE TEMP FUNCTION statements followed by a query.

@tswast tswast added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. api: bigquery Issues related to the BigQuery API. labels Sep 9, 2019
@tswast
Copy link
Contributor Author

tswast commented Sep 10, 2019

There are also some new scripting-related stats (https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobstatistics). For example, numChildJobs and parentJobId.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants