Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect Asset.name when accessing inlet and outlet events #44639

Merged

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Dec 4, 2024

Why

AIP-74 introduces the name attribute to the asset (previously dataset) class, but the outlet_events and inlet_events still only respect the URI attribute.

What

When the users try to access an inlet/outlet event with the asset object, it tries to fetch the asset that matches both the name and URI.

        @task(outlets=[Asset(name="example", uri="test://uri")])
        def produce_asset_events(*, outlet_events):
            outlet_events[Asset(name="example", uri="test://uri")]].extra = {"k": "v"}

before this change, it tries to fetch Asset(uri="test://uri", name="test://uri") instead

close: #44601


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 2 times, most recently from 4e91b11 to 18404ac Compare December 4, 2024 10:05
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 2 times, most recently from a9755cb to eddc6c8 Compare December 4, 2024 14:58
@Lee-W Lee-W marked this pull request as ready for review December 5, 2024 03:21
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 3 times, most recently from 9843519 to 178ecf4 Compare December 5, 2024 08:40
@Lee-W Lee-W changed the title Add attribute.name support to inlet outlet events Respect Asset.name when accessing inlet and outlet events Dec 5, 2024
@Lee-W Lee-W requested a review from uranusjr December 5, 2024 09:08
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 2 times, most recently from 44dd173 to 411362d Compare December 6, 2024 04:27
airflow/utils/context.py Outdated Show resolved Hide resolved
airflow/utils/context.py Outdated Show resolved Hide resolved
airflow/utils/context.py Outdated Show resolved Hide resolved
airflow/utils/context.py Outdated Show resolved Hide resolved
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch from 411362d to 7140d60 Compare December 9, 2024 09:42
airflow/utils/context.py Outdated Show resolved Hide resolved
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch from 5ea5eac to 8183004 Compare December 9, 2024 09:50
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch from f047a64 to e9b0f3c Compare December 10, 2024 10:32
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 3 times, most recently from 58c7370 to 2ae0335 Compare December 10, 2024 13:32
@Lee-W Lee-W requested a review from uranusjr December 10, 2024 14:37
@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch 2 times, most recently from 031d8ed to d3bcd7c Compare December 11, 2024 02:57
airflow/utils/context.py Outdated Show resolved Hide resolved
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor thing. Should be ready for merge otherwise.

@Lee-W Lee-W force-pushed the add-attribute-name-support-to-inlet-outlet-events branch from d3bcd7c to 3054e68 Compare December 11, 2024 07:21
@Lee-W Lee-W merged commit 4289051 into apache:main Dec 11, 2024
97 checks passed
@Lee-W Lee-W deleted the add-attribute-name-support-to-inlet-outlet-events branch December 11, 2024 08:19
ellisms pushed a commit to ellisms/airflow that referenced this pull request Dec 13, 2024
* docs: removing examples that accessing inlet and outlet events

* feat(utils/context): Deprecate accessing inlet and outlet events through string

* fix(util/context): use asset unique key in outlet event accessors

* feat(asset): extend AssetAliasEvent to include asset uri and asset name

* refactor(asset): merge dest_asset_name and dest_asset_uri to dest_asset_key for AssetAliasEvent

* refactor(context): fix context typing

* feat(utils/context): add back string access to inlet and outlet events

* feat(utils/context): forbid using string to access inlet and outlet event

* feat(utils/context): rename AssetUniqueKey and AssetAliasUniqueKey to_obj methods to to_asset, to_asset_alias

* refactor(utils/context): remove redundant iter call

Co-authored-by: Tzu-ping Chung <[email protected]>

* feat(metadata): remove string access support to Metadata

* fix(serialization): fix asset unique key serialization

* feat(asset): make asset unique key an attrs

* fix(serialization): fix asset unique and asset alias unique serialization

* feat(utils/context): change outlet_event error to TypeError when non-asset or non-asset-alias is passed

---------

Co-authored-by: Tzu-ping Chung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Respect asset name attribute when accessing inlet and outlet events
2 participants