Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(providers/common/compat): add back add_input_dataset and add_output_dataset to NoOpCollector #44681

Merged

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Dec 5, 2024

Why

When the user has airflow < 2.10 installed, a provider that has not yet renamed datasets as assets (e.g., apache-airflow-providers-amazon==8.28.0) and apache-airflow-providers-common-compat>=1.2.1 (with asset rename backward compatibility layer.), the user will get the following hook_lineage_collector after calling get_hook_lineage_collector.

    class NoOpCollector:
        """
        NoOpCollector is a hook lineage collector that does nothing.

        It is used when you want to disable lineage collection.
        """


        def add_input_asset(self, *_, **__):
            pass

        def add_output_asset(self, *_, **__):
            pass

As the provider has not yet been updated to use asset methods, it is still called add_input_dataset, which causes an AttributeError as add_input_dataset does not exist in the returned NoOpCollector

What

add back add_input_dataset and add_output_dataset for backward compat


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W force-pushed the fix-common-provider-asset-rename-support branch 2 times, most recently from 3b2ea34 to 71448f5 Compare December 5, 2024 07:46
…et_methods and test_compat_has_asset_and_dataset_methods
@Lee-W Lee-W force-pushed the fix-common-provider-asset-rename-support branch from 71448f5 to 71f3757 Compare December 5, 2024 08:12
@Lee-W Lee-W marked this pull request as ready for review December 5, 2024 08:14
@Lee-W Lee-W merged commit b9ab0b5 into apache:main Dec 5, 2024
65 checks passed
@Lee-W Lee-W deleted the fix-common-provider-asset-rename-support branch December 5, 2024 08:41
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
…ut_dataset to NoOpCollector (apache#44681)

* fix(providers/common/compat): add back add_input_dataset and add_output_dataset to NoOpCollector

* test(providers/common/compat): add test case test_compat_has_only_asset_methods and test_compat_has_asset_and_dataset_methods
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants