Add task status callback #164
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t it can be called cleanly upstream
…cal DB as well as calling the callback if there is one
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
racheldaniel
reviewed
Feb 6, 2023
racheldaniel
reviewed
Feb 6, 2023
racheldaniel
suggested changes
Feb 6, 2023
Awesome, thanks! Addressed the comments, back over to you 😃 |
racheldaniel
added a commit
that referenced
this pull request
Feb 28, 2023
* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Adds quotes to head installs * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * Removes conditionals in github actions * Removes additional branch-- to be managed separately --------- Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]> Co-authored-by: dichenqiandbt <[email protected]>
dichenqiandbt
added a commit
that referenced
this pull request
Mar 2, 2023
* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Add state controll tests and comments. * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Fix format and nits. * nits * Adds quotes to head installs * fix comment * nits * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * nit * nits * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * format * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Nits * Format --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>
dichenqiandbt
added a commit
that referenced
this pull request
Mar 22, 2023
* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Adds quotes to head installs * Add unit test for filesystem service. * nit * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * format * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Fix comment --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>
dichenqiandbt
added a commit
that referenced
this pull request
Mar 28, 2023
* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Add state controll tests and comments. * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Fix format and nits. * nits * Adds quotes to head installs * fix comment * nits * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * nit * nits * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * format * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Nits * Format * Implement task status * Add try --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this PR?
This is a:
All pull requests from community contributors should target the
mainbranch (default).Description & motivation
Checklist