Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add state controll tests and comments. #177

Merged
merged 58 commits into from
Mar 2, 2023
Merged

Add state controll tests and comments. #177

merged 58 commits into from
Mar 2, 2023

Conversation

dichenqiandbt
Copy link
Contributor

What is this PR?

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Add state controller unittest. Leave some thought about this.

  • Unittest file: I personally prefer to put it together with source code, not putting under test/unit... as it's easier to find, let me know your opinion.
  • Private function, I changed several private function name by adding a prefix _.
  • There are several code structure I think deserves a refactor, I will draft a refactor plan on this later.

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
    • Databricks
    • Spark
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models
  • I have added an entry to CHANGELOG.md

racheldaniel and others added 28 commits December 2, 2022 09:20
* Support partial parsing

* Updates todo
* possible way of intergrating all of the dbt commands

* somewhat working version of a generalized framework

* working version of run, a lot of refactor and better core interface needed

* using some new interface

* remove unused function

* using state for run task

* some clean up

* Resolves merge conflicts (#145)

* Core integration updates (#148)

* Updates state_id usage

* Moves task logic to StateController

* removes hardcoded command

* Initiates logmanager in async function

* Removes old async logic and reinstates python logger for dbt-server

Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel Daniel <[email protected]>
…irements.txt to resolve error handling issue with underlying FastAPI dependency (#149)

* Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency

* Add changelog entry
* possible way of intergrating all of the dbt commands

* somewhat working version of a generalized framework

* working version of run, a lot of refactor and better core interface needed

* using some new interface

* remove unused function

* using state for run task

* some clean up

* Core integration updates (#148)

* Updates state_id usage

* Moves task logic to StateController

* removes hardcoded command

* Removes old async logic and reinstates python logger for dbt-server

* Beginning logic to accept a project path

* Adds project_path storing and cacheing

* Removes prints and fixes caching issue

* removes unused task functions

* adds changie entry

* removes dup code from rebase error

* removes dup code from rebase error

* removes dup code from rebase error

* Adds tests for dbt_entry and preliminary state tests

* Removes unused file

* Copies minimal project to tempdir to avoid writing files

Co-authored-by: Chenyu Li <[email protected]>
Co-authored-by: Chenyu Li <[email protected]>
* Updates async endpoint to use set_profile_name function

* Adds checkfirst flag to avoid table exists error

* Fixes profile name and potential fix for sqlalchemy error

* Adds profile back to command args

* Fixes whitespace

* Adds status endpoint
* Adds sync endpoint and fixes linting

* Adds test for sync dbt entry endpoint

* Fixes formatting

* Adds changie entry
* Add the requests library to the requirements

* Replace each specific task update method with a generic method so that it can be called cleanly upstream

* Update this class to use camel casing

* Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one

* Accept a callback url and pass it to the async command method

* Call the new update task status method where the crud methods were previously called

* Move requests from the dev requirements to requirements

* Return the state ID in addition to the other task fields in the async response

* Remove commented out code

* Specify to retry post requests since it isn't enabled by default

* Update dbt_server/views.py

Co-authored-by: Rachel <[email protected]>

* Rename DBTCommandArgs to DbtCommandArgs

* Add a change log entry

---------

Co-authored-by: Rachel <[email protected]>
* Updates db path to working dir instead of app root

* Solidifies locations that the dbt-server writes to

* Changes back to app root after dbt command run

* Fixes comment
…creating the async task. If not present, create a task ID and use it (#168)
* Fix tests.

* Fix wrong package

* Remove adaptor requirements and skip tests without dependency.

* Fix wrong package name
Base automatically changed from feature/v0.2.0 to main February 28, 2023 20:17
dbt_server/state.py Outdated Show resolved Hide resolved
dbt_server/state.py Outdated Show resolved Hide resolved
dbt_server/state.py Outdated Show resolved Hide resolved
dbt_server/state.py Outdated Show resolved Hide resolved
dbt_server/state.py Outdated Show resolved Hide resolved
dbt_server/state.py Outdated Show resolved Hide resolved
Copy link
Contributor

@racheldaniel racheldaniel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments, otherwise this looks good!

@dichenqiandbt
Copy link
Contributor Author

Thanks Rachel for review!

@dichenqiandbt dichenqiandbt merged commit c525eff into main Mar 2, 2023
@dichenqiandbt dichenqiandbt deleted the dichen/dev_1 branch March 2, 2023 21:44
@dichenqiandbt dichenqiandbt restored the dichen/dev_1 branch March 3, 2023 03:27
@dichenqiandbt dichenqiandbt deleted the dichen/dev_1 branch March 3, 2023 03:29
@dichenqiandbt dichenqiandbt restored the dichen/dev_1 branch March 3, 2023 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants