add OpenLineage support to S3ToRedshiftOperator #41422
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
provider:amazon-aws
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
vincbeck
reviewed
Aug 13, 2024
Co-authored-by: Vincent <[email protected]>
ad-m-ss
reviewed
Aug 13, 2024
vincbeck
approved these changes
Aug 14, 2024
…#41414) * Fix tests/models/test_variable.py for database isolation mode * Review feedback
This PR removes SubDags in favor of TaskGroups fro Airflow 3.0 Subdags have been removed from the following locations: - CLI - API - ``SubDagOperator`` This removal marks the end of Subdag support across all interfaces. Users should transition to using TaskGroups as a more efficient and maintainable alternative. --------- Co-authored-by: Brent Bovenzi <[email protected]>
Previously if we ever needed to get the current span and manipulate it, we needed to name it with leading or trailing underscores to avoid collisions with the span decorator function name, which was a bit awkward.
The exception should be `ExportTaskNotFound` (and not `ExportTaskNotFoundFault`) as recorded here: https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DescribeExportTasks.html The moto version dependency pin is updated to reflect this. Older versions of moto will not succeed this test case, hence the version pin is bumped to 5.0.12 Co-authored-by: Vikram Gaonkar <[email protected]>
…ert in system test (apache#41303) * use bulk upsert in system test * fix bulk_upsert test
apache#41449) The self hosted runners from ASF behave strangely: https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25990 So we need to disable publishing docs.
* Remove deprecated and unused methods / properties on DAG (cherry picked from commit 6bd4f83062151d427dab764bca123ba396eda6c0) * add newsfragment * fix test * fix test * Fix formatting --------- Co-authored-by: Tzu-ping Chung <[email protected]>
* restore python 3.12 support for papermill * update provider_dependencies.json
…r and policy removal (apache#41550) * airflow deprecated settings session_lifetime_days, force_log_out_after and policy removal * news fragment added * news fragment updated * news fragment updated * news fragment updated
* smtp email user and password deprecated config removal * smtp email user and password deprecated config removal * news fragment added * smtp email user and password deprecated config removal
…pache#41488) When running tests for providers with old Airflow versions, we have to make sure to uninstall all packages before installing old Airflow versions - because some of the packages can be installed by new Airflow version and they are missing in the old Airflow version. This has already happened in apache#41402 with methodtools. This PR adds `--clean-airflow-installation` flag to relevant breeze commands that install other airflow version. This is quite a bit slower as it requires to uninstall and reinstall packages so we do not set it by default.
…apache#41096) * processor_poll_interval deprecated parameter usage removal * news fragment added * news fragment added * news fragment added * Update newsfragments/41096.significant.rst Co-authored-by: Jarek Potiuk <[email protected]> * Update 41096.significant.rst --------- Co-authored-by: Jarek Potiuk <[email protected]>
Artuz37
requested review from
jedcunningham,
ephraimbuddy,
dstandish,
kaxil,
pierrejeambrun,
potiuk,
ashb,
mobuchowski,
Fokko,
mik-laj,
hussein-awala,
josh-fell,
uranusjr,
ryanahamilton,
bbovenzi,
bolkedebruin and
XD-DENG
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
get_openlineage_facets_on_completemethod toS3ToRedshiftOperator.It extracts target table schema by querying redshift and creates identity column level lineage.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.