-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor python config handling #830
Conversation
… config file (#724) Signed-off-by: Artur Peedimaa <[email protected]> Co-authored-by: Artur Peedimaa <[email protected]> Co-authored-by: Ben Cassell <[email protected]>
Signed-off-by: Dmitry Volodin <[email protected]> Co-authored-by: Ben Cassell <[email protected]>
Co-authored-by: Antoine <[email protected]>
Signed-off-by: Roy Dobbe [email protected] Co-authored-by: Roy Dobbe <[email protected]>
Signed-off-by: Kyle Valade <[email protected]> Co-authored-by: Kyle Valade <[email protected]> Co-authored-by: Ben Cassell <[email protected]>
``` | ||
|
||
#### Grants | ||
|
||
You might want to give certain users or teams access to run your workflows outside of | ||
dbt in an ad hoc way. You can define those permissions in the `workflow_job_config.grants`. | ||
dbt in an ad hoc way. You can define those permissions in the `python_job_config.grants`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this with find/replace, and VS decided it needed to reformat the whole file :/
Moved to using pydantic for managing config verification/extraction, added factory methods ( @jackyhu-db @eric-wang-1990 @kdazzle apologies for complex refactor, but python_submission.py was already a mess, and Kyle's feature added a bunch of feature support that pushed that module beyond the limits of what I can mentally process. |
Description
Refactoring python submission components to be a little better factored. There's still definitely more cleanup to do, but my goal was basically to get the config manipulation and submission into the 3 core ways we actually submit (Workflow, Job/Run, and Command) so that we can maybe one day move to those being the submission helpers, with compute not being named in the submission method. For now we have to keep the top-level submission helpers so that we don't break any existing users.
Discussed with original author Kyle on renaming 'workflow_job_config' to 'python_job_config' so as to not confuse users when using this config without the workflow submission method.
Checklist
CHANGELOG.md
and added information about my change to the "dbt-databricks next" section.