-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix CI issues #1005
Merged
Fix CI issues #1005
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
captured by test FAILED tests/operators/test_local.py::test_run_operator_dataset_emission_fails - Failed: DID NOT RAISE <class 'cosmos.exceptions.AirflowCompatibilityError'>
…st version of the Apache Airflow providers: apache/airflow#39842 apache-airflow-providers-postgres==5.11.1rc1
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1005 +/- ##
==========================================
- Coverage 95.72% 95.65% -0.07%
==========================================
Files 60 60
Lines 2926 2925 -1
==========================================
- Hits 2801 2798 -3
- Misses 125 127 +2 ☔ View full report in Codecov by Sentry. |
dosubot
bot
added
size:M
This PR changes 30-99 lines, ignoring generated files.
area:ci
Related to CI, Github Actions, or other continuous integration tools
area:dependencies
Related to dependencies, like Python packages, library versions, etc
profile:postgres
Related to Postgres ProfileConfig
labels
May 28, 2024
tatiana
commented
May 29, 2024
"apache-airflow-providers-cncf-kubernetes>=5.1.1", | ||
"apache-airflow-providers-amazon>=3.0.0,<8.20.0", # https://github.com/apache/airflow/issues/39103 | ||
"apache-airflow-providers-docker>=3.5.0", | ||
"apache-airflow-providers-microsoft-azure", | ||
"apache-airflow-providers-postgres", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bring back to alphabetic order
pankajkoti
approved these changes
May 29, 2024
pankajastro
approved these changes
May 29, 2024
tatiana
added a commit
that referenced
this pull request
Jun 6, 2024
Fix three problems observed while running the tests in the CI recently: 1. Static check Fix spelling issue captured by static check (#1000) 2. Integration test that's no longer needed The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider `apache-airflow-providers-postgres==5.11.1rc1`: apache/airflow#39842 3. Skip a buggy version of OL There was a breaking change between `openlineage-integration-common==1.14.0` and `openlineage-integration-common==1.15.0` . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693. This is an example of how Cosmos was using `DbtLocalArtifactProcessor` - something that had been agreed upon in the past: ``` openlineage_processor = DbtLocalArtifactProcessor( producer=OPENLINEAGE_PRODUCER, job_namespace=LINEAGE_NAMESPACE, project_dir=project_dir, profile_name=self.profile_config.profile_name, target=self.profile_config.target_name, ) events = openlineage_processor.parse() for completed in events.completes: for output in getattr(completed, source): dataset_uri = output.namespace + "/" + output.name uris.append(dataset_uri) ``` In `openlineage-integration-common==1.14.0` and earlier versions, this would create URIs in the format: ``` postgres://0.0.0.0:5432/postgres.public.stg_customers ``` Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format: ``` postgres.public.stg_customers/postgres://0.0.0.0:5432 ``` This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16: https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
Merged
tatiana
added a commit
that referenced
this pull request
Jun 6, 2024
Bug fixes * Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663 in #1023 * Fix Cosmos ``enable_cache`` setting by @tatiana in #1025 * Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile arg optional by @oliverrmaa and @pankajastro in #839 and #1017 * Athena profile mapping set ``aws_session_token`` in profile only if it exists by @pankajastro in #1022 Others * Update dbt and Airflow conflicts matrix by @tatiana in #1026 * Enable Python 3.12 unittest by @pankajastro in #1018 * Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in #1004 * Add GitHub issue templates for bug reports and feature request by @pankajkoti in #1009 * Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in #1027 * Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time by @dwreeves in #997 * Drop support for Airflow 2.3 by @pankajkoti in #994 * Update Astro Runtime image by @RNHTTR in #988 and #989 * Enable ruff F linting by @pankajastro in #985 * Move Cosmos Airflow configuration to settings.py by @pankajastro in #975 * Fix CI Issues by @tatiana in #1005 * Pre-commit hook updates in #1000, #1019 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Reeves <[email protected]> Co-authored-by: Pankaj Koti <[email protected]> Co-authored-by: davidsteinar <[email protected]> Co-authored-by: David Steinar Asgrimsson <[email protected]> Co-authored-by: Marco Yuen <[email protected]> Co-authored-by: Pankaj Singh <[email protected]> Co-authored-by: Ollie Ma <[email protected]>
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Fix three problems observed while running the tests in the CI recently: 1. Static check Fix spelling issue captured by static check (astronomer#1000) 2. Integration test that's no longer needed The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider `apache-airflow-providers-postgres==5.11.1rc1`: apache/airflow#39842 3. Skip a buggy version of OL There was a breaking change between `openlineage-integration-common==1.14.0` and `openlineage-integration-common==1.15.0` . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693. This is an example of how Cosmos was using `DbtLocalArtifactProcessor` - something that had been agreed upon in the past: ``` openlineage_processor = DbtLocalArtifactProcessor( producer=OPENLINEAGE_PRODUCER, job_namespace=LINEAGE_NAMESPACE, project_dir=project_dir, profile_name=self.profile_config.profile_name, target=self.profile_config.target_name, ) events = openlineage_processor.parse() for completed in events.completes: for output in getattr(completed, source): dataset_uri = output.namespace + "/" + output.name uris.append(dataset_uri) ``` In `openlineage-integration-common==1.14.0` and earlier versions, this would create URIs in the format: ``` postgres://0.0.0.0:5432/postgres.public.stg_customers ``` Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format: ``` postgres.public.stg_customers/postgres://0.0.0.0:5432 ``` This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16: https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Bug fixes * Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663 in astronomer#1023 * Fix Cosmos ``enable_cache`` setting by @tatiana in astronomer#1025 * Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile arg optional by @oliverrmaa and @pankajastro in astronomer#839 and astronomer#1017 * Athena profile mapping set ``aws_session_token`` in profile only if it exists by @pankajastro in astronomer#1022 Others * Update dbt and Airflow conflicts matrix by @tatiana in astronomer#1026 * Enable Python 3.12 unittest by @pankajastro in astronomer#1018 * Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in astronomer#1004 * Add GitHub issue templates for bug reports and feature request by @pankajkoti in astronomer#1009 * Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in astronomer#1027 * Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time by @dwreeves in astronomer#997 * Drop support for Airflow 2.3 by @pankajkoti in astronomer#994 * Update Astro Runtime image by @RNHTTR in astronomer#988 and astronomer#989 * Enable ruff F linting by @pankajastro in astronomer#985 * Move Cosmos Airflow configuration to settings.py by @pankajastro in astronomer#975 * Fix CI Issues by @tatiana in astronomer#1005 * Pre-commit hook updates in astronomer#1000, astronomer#1019 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Reeves <[email protected]> Co-authored-by: Pankaj Koti <[email protected]> Co-authored-by: davidsteinar <[email protected]> Co-authored-by: David Steinar Asgrimsson <[email protected]> Co-authored-by: Marco Yuen <[email protected]> Co-authored-by: Pankaj Singh <[email protected]> Co-authored-by: Ollie Ma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:ci
Related to CI, Github Actions, or other continuous integration tools
area:dependencies
Related to dependencies, like Python packages, library versions, etc
lgtm
This PR has been approved by a maintainer
profile:postgres
Related to Postgres ProfileConfig
size:M
This PR changes 30-99 lines, ignoring generated files.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix three problems observed while running the tests in the CI recently:
Fix spelling issue captured by static check (#1000)
The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider
apache-airflow-providers-postgres==5.11.1rc1
: apache/airflow#39842There was a breaking change between
openlineage-integration-common==1.14.0
andopenlineage-integration-common==1.15.0
. It may have been an unintended side-effect of OpenLineage/OpenLineage#2693.This is an example of how Cosmos was using
DbtLocalArtifactProcessor
- something that had been agreed upon in the past:In
openlineage-integration-common==1.14.0
and earlier versions, this would create URIs in the format:Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format:
This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0