[CLOUDEXEC-766] Add callback function. #203
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Support partial parsing * Updates todo
…server into feature/server-on-core-1.5
* possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]>
…irements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry
* possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]>
* Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint
* Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry
* Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]>
* Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment
…creating the async task. If not present, create a task ID and use it (#168)
* Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name
* Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR
* Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix
* Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format.
… into dichen/dev_1
Co-authored-by: Rachel <[email protected]>
…into dichen/dev_1
Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel <[email protected]>
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
| @@ -97,7 +111,8 @@ def _invoke(task: Any, command: List[str], callback_url: Optional[str] = None): | |||
| task_id = task.request.id | |||
| _insert_log_path(command, task_id) | |||
| logger.info(f"Running dbt task ({task_id}) with {command}") | |||
| # TODO: Send callback to infer task start. | |||
| if callback_url: | |||
| _send_state_callback(callback_url, task_id, STARTED) | |||
There was a problem hiding this comment.
So at this point the task would technically show PENDING if we called for the status, right? I see how this gives the caller more accurate information, I'm just curious as to your thinking around sending this here rather than doing a _get_task_status() after we call t.start(). Is there a chance we tell a client the task has started and then it actually fails to start?
There was a problem hiding this comment.
Actually it shows STARTED not PENDING. PENDING = task is created buy still wait for worker process to pickup. STARTED = task has been picked up by worker. STARTED is automatically set by celery as I passed
track_started=True.
But you gave me a very good hint! I should try catch in callback helper to make sure it doesn't halt worker.
There was a problem hiding this comment.
Oh cool, thanks so much!
racheldaniel
approved these changes
Mar 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this PR?
This is a:
All pull requests from community contributors should target the
mainbranch (default).Description & motivation
Added callback function in celery task, it will be triggered once state is updated.
Checklist