Update actions #176

racheldaniel · 2023-02-23T21:20:12Z

What is this PR?

This is a:

documentation update
bug fix with no breaking changes
new functionality
a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Checklist

dichenqiandbt · 2023-02-23T22:09:36Z

.github/workflows/main.yml

    steps:
      - name: checkout repo
+        if: ${{ matrix.dbt-core.server-branch == github.ref_name || matrix.dbt-core.server-branch == github.event.pull_request.base.ref }}


I noticed this condition and first want to learn from you it's meaning. matrix.dbt-core.server-branch == github.ref_name is for push action while github.event.pull_request.base.ref is for pull request action.

Secondly can we use job level if instead of step level if? So we don't need to duplicate condition three times.
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idif

The conditional just means to only run if it's on the server-branch or if it's a PR to that server-branch -- so yes, the first will only evaluate to True on a push and the second will evaluate to True on a PR.

To your second point-- yes! I didn't realize job-level was possible, thank you for finding that 👍

Oh! I'm wrong on the second point, I know why this isn't job level -- this conditional won't have access to the matrix and those values until it starts looping through them

Thanks Rachel! I found this actions/runner#1985 about if matrix context works in job level "if", seems like lot's of folks complained about it...

While I'm spending some times digging into this, maybe I find another workaround using env for better readability. We can define env under job, like IS_INTERESTED_SERVER_BRANCH those envs can be reused below so we sync the logic between three steps.

Let me know your thoughts on this! Do you think it would be better from readability perspective? My feeling is that it's easier to understand. I don't have strong opinion on this, just because I don't like repeated code pieces.

strategy:
matrix:
color: ["red", "pink"]
env:
COLOR_ENV: ${{ matrix.color == 'pink' }}
steps:
- if: ${{ env.COLOR_ENV == 'true' }}
See run results in https://github.com/dichenqiandbt/test-dbt.

I don't think I quite understand how the env value is being set in your example-- are you defining that in the repo settings? I can see in the actions how it's conditionally running actions based on that env value, but where can I see what the conditional actually is? I have no problem DRYing up the code as long as it's still easy to tell what we're building with each branch!

It's also an option for this actions file to be different for different branches, and I could leave off all of the conditionals. I thought it might be better to be explicit, but I don't have strong feelings either 🙂

@peter-bertuglia Curious for your thoughts here as well-- these actions are a bit messy, I'm down for whatever is clearest!

.github/workflows/main.yml

* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Adds quotes to head installs * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * Removes conditionals in github actions * Removes additional branch-- to be managed separately --------- Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]> Co-authored-by: dichenqiandbt <[email protected]>

* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Add state controll tests and comments. * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Fix format and nits. * nits * Adds quotes to head installs * fix comment * nits * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * nit * nits * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * format * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Nits * Format --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>

* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Adds quotes to head installs * Add unit test for filesystem service. * nit * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * format * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/services/filesystem_service.py Co-authored-by: Rachel <[email protected]> * Fix comment --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>

* Fixes legacy logger * Initial pass at removing legacy logging * Update log file output to use json format * Updates server logging to send events * Removes outdated comment * Support partial parsing (#151) * Support partial parsing * Updates todo * Core integration playground (#129) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Resolves merge conflicts (#145) * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Initiates logmanager in async function * Removes old async logic and reinstates python logger for dbt-server Co-authored-by: Rachel <[email protected]> Co-authored-by: Rachel Daniel <[email protected]> * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149) * Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency * Add changelog entry * Accept project path in addition to state_id (#154) * possible way of intergrating all of the dbt commands * somewhat working version of a generalized framework * working version of run, a lot of refactor and better core interface needed * using some new interface * remove unused function * using state for run task * some clean up * Core integration updates (#148) * Updates state_id usage * Moves task logic to StateController * removes hardcoded command * Removes old async logic and reinstates python logger for dbt-server * Beginning logic to accept a project path * Adds project_path storing and cacheing * Removes prints and fixes caching issue * removes unused task functions * adds changie entry * removes dup code from rebase error * removes dup code from rebase error * removes dup code from rebase error * Adds tests for dbt_entry and preliminary state tests * Removes unused file * Copies minimal project to tempdir to avoid writing files Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Chenyu Li <[email protected]> * Fixes broken tests (#156) * Fix profile for async endpoint (#157) * Updates async endpoint to use set_profile_name function * Adds checkfirst flag to avoid table exists error * Fixes profile name and potential fix for sqlalchemy error * Adds profile back to command args * Fixes whitespace * Adds status endpoint * Fixes shutdown and removes middleware * Fixes response_model as called out by community member on main branch * Sync dbt endpoint (#161) * Adds sync endpoint and fixes linting * Adds test for sync dbt entry endpoint * Fixes formatting * Adds changie entry * Add task status callback (#164) * Add the requests library to the requirements * Replace each specific task update method with a generic method so that it can be called cleanly upstream * Update this class to use camel casing * Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one * Accept a callback url and pass it to the async command method * Call the new update task status method where the crud methods were previously called * Move requests from the dev requirements to requirements * Return the state ID in addition to the other task fields in the async response * Remove commented out code * Specify to retry post requests since it isn't enabled by default * Update dbt_server/views.py Co-authored-by: Rachel <[email protected]> * Rename DBTCommandArgs to DbtCommandArgs * Add a change log entry --------- Co-authored-by: Rachel <[email protected]> * make server working with dbt-core main (#167) * Control server write locations (#166) * Updates db path to working dir instead of app root * Solidifies locations that the dbt-server writes to * Changes back to app root after dbt command run * Fixes comment * Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168) * Adds error handling for json conversion * Include all exceptions in error handling. (#169) * Fix bug of not chdir back (#175) * Fix tests. (#173) * Fix tests. * Fix wrong package * Remove adaptor requirements and skip tests without dependency. * Fix wrong package name * Add state controll tests and comments. * Update actions (#176) * Resolves merge conflicts * Cherry-pick gone awry * spaces * Allows images to build on PR * Removes conditional on test, tailors to branch (#181) * Removes conditional on test, tailors to branch * Adds changie entry * Comments out unused matrix * fixes formatting * fixes formatting * Testing installations in one line * Undoes consolidaiton to single line * Fix format and nits. * nits * Adds quotes to head installs * fix comment * nits * Adds 1.5.0b1 to github action * Adds prerelease flag * RUNTIME-733 Add smoke tests (#170) * Add smoke test and check in test dbt project jaffle shop. * nits * nits * Add changie * Update smoke test. * Start dbt-server inside smoke test. * Fix format. * Fixes linting * nit * nits * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * format * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Update dbt_server/state.py Co-authored-by: Rachel <[email protected]> * Nits * Format * Implement task status * Add try --------- Co-authored-by: Rachel Daniel <[email protected]> Co-authored-by: Rachel <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: Jennifer Miller <[email protected]> Co-authored-by: Chenyu Li <[email protected]> Co-authored-by: jp-dbt <[email protected]>

racheldaniel added 2 commits February 23, 2023 15:11

Resolves merge conflicts

e213acc

Cherry-pick gone awry

8b62fa7

cla-bot bot added the cla:yes label Feb 23, 2023

spaces

82af53e

racheldaniel requested review from peter-bertuglia and dichenqiandbt February 23, 2023 21:33

dichenqiandbt reviewed Feb 23, 2023

View reviewed changes

.github/workflows/main.yml Show resolved Hide resolved

dichenqiandbt reviewed Feb 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Allows images to build on PR

9db745d

dichenqiandbt approved these changes Feb 24, 2023

View reviewed changes

peter-bertuglia approved these changes Feb 24, 2023

View reviewed changes

racheldaniel merged commit 6797958 into feature/v0.2.0 Feb 27, 2023

racheldaniel deleted the racheld/RUNTIME-715/update-actions branch February 27, 2023 15:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update actions #176

Update actions #176

racheldaniel commented Feb 23, 2023 •

edited

Loading

dichenqiandbt Feb 23, 2023

racheldaniel Feb 24, 2023 •

edited

Loading

racheldaniel Feb 24, 2023 •

edited

Loading

dichenqiandbt Feb 24, 2023 •

edited

Loading

racheldaniel Feb 24, 2023

racheldaniel Feb 24, 2023

Update actions #176

Update actions #176

Conversation

racheldaniel commented Feb 23, 2023 • edited Loading

What is this PR?

Description & motivation

Checklist

dichenqiandbt Feb 23, 2023

Choose a reason for hiding this comment

racheldaniel Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

racheldaniel Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

dichenqiandbt Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

racheldaniel Feb 24, 2023

Choose a reason for hiding this comment

racheldaniel Feb 24, 2023

Choose a reason for hiding this comment

racheldaniel commented Feb 23, 2023 •

edited

Loading

racheldaniel Feb 24, 2023 •

edited

Loading

racheldaniel Feb 24, 2023 •

edited

Loading

dichenqiandbt Feb 24, 2023 •

edited

Loading