Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow depending to a @task_group as a whole #20671

Merged
merged 1 commit into from
Jan 8, 2022

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Jan 5, 2022

Fix #19903.

Instead of always using the task returned by the task group function as the dependency representation, also allow a task group function to not return any task specifically.

Setting dependency to such a group would depend on the group's contained tasks as a whole, and not any specific task in it.

This renders this DAG definition taken from #19903 (comment) (which would fail to parse currently)

@dag(start_date=datetime(2023, 1, 1), schedule_interval="@once")
def test_dag_1():
    @task
    def start():
        pass

    @task
    def do_thing(x):
        print(x)

    @task_group
    def do_all_things():
        do_thing(1)
        do_thing(2)

    @task
    def end():
        pass

    start() >> do_all_things() >> end()

my_dag = test_dag_1()

like this:

image

Tests to come later. Done

@uranusjr uranusjr force-pushed the taskflow-task-group-dependency branch from d45cfb3 to 8a89cea Compare January 5, 2022 11:12
@uranusjr uranusjr marked this pull request as ready for review January 5, 2022 11:13
@mik-laj mik-laj requested a review from turbaszek January 5, 2022 16:43
Copy link
Contributor

@josh-fell josh-fell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love that this gets to parity with the context manager version of TaskGroups!

airflow/decorators/task_group.py Outdated Show resolved Hide resolved
Instead of always using the task returned by the task group function as
the dependency representation, also allow a task group function to *not*
return any task specifically.

Setting dependency to such a group would depend on the group's contained
tasks as a whole.
@uranusjr uranusjr force-pushed the taskflow-task-group-dependency branch from 3b3573c to 3f5a523 Compare January 7, 2022 05:09
Copy link
Contributor

@eladkal eladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@github-actions
Copy link

github-actions bot commented Jan 7, 2022

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jan 7, 2022
@josh-fell
Copy link
Contributor

Doesn't need to be in this PR but it might be nice to have some documentation on these two behaviors with the @task_group decorator.

@uranusjr I can log a request for this if you don't want to take this up. Let me know.

@josh-fell
Copy link
Contributor

Actually looks like this would be all new documentation on the @task_group decorator. I can't find any docs in 2.2.3 that outline the feature (at least not easily) and no docs seem to reference the [START/END howto_task_group_decorator] markers in the example_task_group_decorator DAG too.

@uranusjr
Copy link
Member Author

uranusjr commented Jan 8, 2022

I created #20762 to track documentation changes.

@uranusjr uranusjr merged commit 384fa4a into apache:main Jan 8, 2022
@uranusjr uranusjr deleted the taskflow-task-group-dependency branch January 8, 2022 04:09
@potiuk potiuk modified the milestone: Airflow 2.2.4 Jan 31, 2022
@eladkal eladkal added this to the Airflow 2.3.0 milestone Jan 31, 2022
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Apr 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error setting dependencies on task_group defined using the decorator
5 participants