Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor python config handling #830

Merged
merged 41 commits into from
Oct 22, 2024
Merged

Refactor python config handling #830

merged 41 commits into from
Oct 22, 2024

Conversation

benc-db
Copy link
Collaborator

@benc-db benc-db commented Oct 17, 2024

Description

Refactoring python submission components to be a little better factored. There's still definitely more cleanup to do, but my goal was basically to get the config manipulation and submission into the 3 core ways we actually submit (Workflow, Job/Run, and Command) so that we can maybe one day move to those being the submission helpers, with compute not being named in the submission method. For now we have to keep the top-level submission helpers so that we don't break any existing users.

Discussed with original author Kyle on renaming 'workflow_job_config' to 'python_job_config' so as to not confuse users when using this config without the workflow submission method.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

benc-db and others added 30 commits June 24, 2024 13:24
… config file (#724)

Signed-off-by: Artur Peedimaa <[email protected]>
Co-authored-by: Artur Peedimaa <[email protected]>
Co-authored-by: Ben Cassell <[email protected]>
Signed-off-by: Dmitry Volodin <[email protected]>
Co-authored-by: Ben Cassell <[email protected]>
Signed-off-by: Roy Dobbe [email protected]
Co-authored-by: Roy Dobbe <[email protected]>
Signed-off-by: Kyle Valade <[email protected]>
Co-authored-by: Kyle Valade <[email protected]>
Co-authored-by: Ben Cassell <[email protected]>
```

#### Grants

You might want to give certain users or teams access to run your workflows outside of
dbt in an ad hoc way. You can define those permissions in the `workflow_job_config.grants`.
dbt in an ad hoc way. You can define those permissions in the `python_job_config.grants`.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this with find/replace, and VS decided it needed to reformat the whole file :/

@benc-db
Copy link
Collaborator Author

benc-db commented Oct 18, 2024

Moved to using pydantic for managing config verification/extraction, added factory methods (create) to simplify init methods. I don't like having complex logic in inits, because it makes unit testing with mocks waaay harder, but adding the factory methods make it much cleaner to build the object graphs...really wish I could pass the dependencies into the helpers, but we don't own the instantiation (happens inside dbt). Will move onto adding/updating unit tests now.

@jackyhu-db @eric-wang-1990 @kdazzle apologies for complex refactor, but python_submission.py was already a mess, and Kyle's feature added a bunch of feature support that pushed that module beyond the limits of what I can mentally process.

jackyhu-db
jackyhu-db previously approved these changes Oct 22, 2024
@benc-db benc-db merged commit 84fc024 into main Oct 22, 2024
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants