Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support building DbtDag without setting paths in ProjectConfig #1307

Merged
merged 1 commit into from
Nov 6, 2024

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Nov 6, 2024

Previously, users were unable to run a DbtDag or DbtTask group where the following was used:

  • RenderMode.DBT_LS
  • ExecutionConfig(dbt_project_path)
  • RenderConfig(dbt_project_path)

This scenario can be helpful when using ExecutionMode.KUBERNETES or other similar ones and was found out during: #1297

…onfig

Previously, users were unable to run a DbtDag or DbtTask group where the following was used:
- RenderMode.DBT_LS
- ExecutionConfig(dbt_project_path)
- RenderConfig(dbt_project_path)

This scenario can be useful when using ExecutionMode.KUBERNETES or other similar ones and was found out during:
#1297
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Nov 6, 2024
@dosubot dosubot bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc labels Nov 6, 2024
Copy link

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3cf717e
Status: ✅  Deploy successful!
Preview URL: https://6727bd53.astronomer-cosmos.pages.dev
Branch Preview URL: https://fix-k8s-with-renderconfig-db.astronomer-cosmos.pages.dev

View logs

@tatiana tatiana changed the title Allow DbtToAirflowConverter to work without ProjectConfig paths Support building DbtDag without setting paths in ProjectConfig Nov 6, 2024
@tatiana
Copy link
Collaborator Author

tatiana commented Nov 6, 2024

There are two checks currently failing in this PR:

  1. Related to the existing latency issue when building (3.8, 2.6) dependencies, which is addressed in [CI] Reduce the amount taking to run tests in the CI from 5h to 11min #1297
  2. Issue related to a dbt conflict with the protobuf version:
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/venv/lib/python3.11/site-packages/dbt_common/events/base_types.py", line 139, in msg_from_base_event
    "msg": event.message(),
           ^^^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/venv/lib/python3.11/site-packages/dbt/events/types.py", line 2059, in message
    return f"Resource report: {self.to_json()}"
                               ^^^^^^^^^^^^^^
  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/venv/lib/python3.11/site-packages/dbt_common/events/base_types.py", line 98, in to_json
    return MessageToJson(
           ^^^^^^^^^^^^^^
TypeError: MessageToJson() got an unexpected keyword argument 'including_default_value_fields'

dbt-labs/dbt-core#9759

The current PR causes neither issue.

I rebased PR #1297 using this branch and confirmed both issues are resolved in: https://github.com/astronomer/astronomer-cosmos/actions/runs/11706266140

Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for finding the issue, isolating & creating a PR for this. Looks robust now.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 6, 2024
@tatiana tatiana merged commit 739db95 into main Nov 6, 2024
59 of 62 checks passed
@tatiana tatiana deleted the fix-k8s-with-renderconfig-dbt-ls branch November 6, 2024 16:36
@tatiana tatiana added this to the Cosmos 1.8.0 milestone Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants