Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix running child tasks in a subdag after clearing a successful subdag #14776

Merged
merged 4 commits into from
Mar 18, 2021
Merged

Fix running child tasks in a subdag after clearing a successful subdag #14776

merged 4 commits into from
Mar 18, 2021

Conversation

ephraimbuddy
Copy link
Contributor

Closes: #13295

After successfully running a SUBDAG, clearing it
(including downstream+recursive) doesn't trigger the inner tasks.
Instead, the subdag is marked successful and the inner tasks all
stay cleared and aren't re-run.

The above problem is because the DagRun state of the subdags are not updated
after clearing. This PR solves it by updating the DagRun state of all DAGs
including subdags when include_subdags is True

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@ephraimbuddy ephraimbuddy force-pushed the fix-child-task-not-run-in-clear-subdag branch from 83d1b8d to cd21515 Compare March 15, 2021 08:12
kaxil
kaxil reviewed Mar 15, 2021
View reviewed changes
airflow/models/dag.py Outdated Show resolved Hide resolved
@ephraimbuddy ephraimbuddy force-pushed the fix-child-task-not-run-in-clear-subdag branch from cd21515 to a6eac67 Compare March 15, 2021 16:26
ashb
ashb reviewed Mar 16, 2021
View reviewed changes
tests/models/test_dag.py Outdated Show resolved Hide resolved
ashb
ashb reviewed Mar 16, 2021
View reviewed changes
airflow/models/dag.py Outdated Show resolved Hide resolved
ashb
ashb reviewed Mar 16, 2021
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test isn't right yet -- it's passing on master so isn't correctly exercising the bug

@ephraimbuddy ephraimbuddy force-pushed the fix-child-task-not-run-in-clear-subdag branch from a6eac67 to 1bb5e65 Compare March 16, 2021 11:32
@ephraimbuddy ephraimbuddy requested review from ashb and kaxil March 16, 2021 11:33
@ephraimbuddy
Copy link
Contributor Author

This test isn't right yet -- it's passing on master so isn't correctly exercising the bug

Resolved. Thanks for info on testing!!!

@kaxil kaxil added this to the Airflow 2.0.2 milestone Mar 17, 2021
@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Mar 17, 2021
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@ephraimbuddy
Fix running child tasks in a subdag after clearing a successful subdag
703e652 
After successfully running a SUBDAG, clearing it
(including downstream+recursive) doesn't trigger the inner tasks.
Instead, the subdag is marked successful and the inner tasks all
stay cleared and aren't re-run.

The above problem is because the DagRun state of the subdags are not updated
after clearing. This PR solves it by updating the DagRun state of all DAGs
including subdags when include_subdags is True
@ephraimbuddy
fixup! Fix running child tasks in a subdag after clearing a successfu…
4e953a1 
…l subdag
@ephraimbuddy
fixup! fixup! Fix running child tasks in a subdag after clearing a su…
741fa56 
…ccessful subdag
@ephraimbuddy
Apply suggestions from code review and fix test
cec7ef2
@ephraimbuddy ephraimbuddy force-pushed the fix-child-task-not-run-in-clear-subdag branch from 1bb5e65 to cec7ef2 Compare March 17, 2021 23:36
@ashb ashb merged commit 0521635 into apache:master Mar 18, 2021
@ashb ashb deleted the fix-child-task-not-run-in-clear-subdag branch March 18, 2021 10:38
ashb pushed a commit that referenced this pull request Mar 19, 2021
@ephraimbuddy @ashb
Fix running child tasks in a subdag after clearing a successful subdag (
f4cc5c5 
#14776)

After successfully running a SUBDAG, clearing it
(including downstream+recursive) doesn't trigger the inner tasks.
Instead, the subdag is marked successful and the inner tasks all
stay cleared and aren't re-run.

The above problem is because the DagRun state of the subdags are not updated
after clearing. This PR solves it by updating the DagRun state of all DAGs
including subdags when include_subdags is True

(cherry picked from commit 0521635)
@ephraimbuddy ephraimbuddy mentioned this pull request Apr 15, 2021
Closed
@zarrarrana zarrarrana mentioned this pull request Apr 28, 2021
Merged
kaxil pushed a commit that referenced this pull request Apr 30, 2021
Fix changing the parent dag state on subdag clear (#15562)
a4211e2 
Closes: #15374
This pull request follows #14776. 

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.
potiuk pushed a commit that referenced this pull request May 9, 2021
@potiuk
Fix changing the parent dag state on subdag clear (#15562)
aa58a0a 
Closes: #15374
This pull request follows #14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

(cherry picked from commit a4211e2)
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 17, 2021
Fix changing the parent dag state on subdag clear (#15562)
f7c3e8d 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
@nsepetys nsepetys mentioned this pull request Sep 21, 2021
Closed
2 tasks
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 23, 2021
Fix changing the parent dag state on subdag clear (#15562)
f430f96 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Nov 27, 2021
Fix changing the parent dag state on subdag clear (#15562)
1849ee6 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Mar 10, 2022
Fix changing the parent dag state on subdag clear (#15562)
1d3d155 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 4, 2022
Fix changing the parent dag state on subdag clear (#15562)
e163f73 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jul 9, 2022
Fix changing the parent dag state on subdag clear (#15562)
6722a14 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Aug 27, 2022
Fix changing the parent dag state on subdag clear (#15562)
4f4cc0e 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 4, 2022
Fix changing the parent dag state on subdag clear (#15562)
feaf7ae 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
aglipska pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 7, 2022
Fix changing the parent dag state on subdag clear (#15562)
db138b6 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Dec 7, 2022
Fix changing the parent dag state on subdag clear (#15562)
0c57d6f 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jan 27, 2023
Fix changing the parent dag state on subdag clear (#15562)
1cac4d4 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
kosteev pushed a commit to kosteev/composer-airflow-test-copybara that referenced this pull request Sep 12, 2024
Fix changing the parent dag state on subdag clear (#15562)
5f447f6 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
kosteev pushed a commit to kosteev/composer-airflow-test-copybara that referenced this pull request Sep 13, 2024
Fix changing the parent dag state on subdag clear (#15562)
c86afa7 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 17, 2024
Fix changing the parent dag state on subdag clear (#15562)
c6819b2 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Nov 7, 2024
Fix changing the parent dag state on subdag clear (#15562)
1f3996f 
Closes: apache/airflow#15374
This pull request follows apache/airflow#14776.

Clearing a subdag with Downstream+Recursive does not automatically set the state of the parent dag so that the downstream parent tasks can execute.

GitOrigin-RevId: a4211e276fce6521f0423fe94b01241a9c43a22c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil approved these changes

@ashb ashb ashb approved these changes

@turbaszek turbaszek Awaiting requested review from turbaszek

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

Assignees
No one assigned
Labels
full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Milestone
Airflow 2.0.2
Development

Successfully merging this pull request may close these issues.

In triggered SubDag (schedule_interval=None), when clearing a successful Subdag, child tasks aren't run
3 participants
@ephraimbuddy @ashb @kaxil