Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mark tasks as failed when using ONE_SUCCESS and Upstream has no successes #15467

Merged
merged 1 commit into from
May 4, 2021
Merged

Mark tasks as failed when using ONE_SUCCESS and Upstream has no successes #15467

merged 1 commit into from
May 4, 2021

Conversation

r-richmond
Copy link
Contributor

@r-richmond r-richmond commented Apr 21, 2021 •
edited
Loading

What this does

  1. When tasks use trigger_rule = ONE_SUCCESS Mark them as upstream_failed when no upstreams succeed
    • Leaves skipped behavior when all upstream are skipped
  2. Adds several new tests to ensure the new behavior is as expected

Why do this

  1. I had several dags that weren't failing out as expected when no successes were found
  2. Other users appear to have run into this same issue https://stackoverflow.com/questions/64603004/aiflow-skipping-task-on-one-success-trigger-rule

@kaxil kaxil closed this Apr 21, 2021
@kaxil kaxil reopened this Apr 21, 2021
@kaxil
Copy link
Member

kaxil commented Apr 21, 2021

Closed/reopened to trigger CI job again

@r-richmond r-richmond changed the title Mark tasks as failed when using ONE_SUCCESS and Upstream fails to have one success Mark tasks as failed when using ONE_SUCCESS and Upstream has no successes Apr 21, 2021
@r-richmond r-richmond closed this Apr 22, 2021
@r-richmond r-richmond reopened this Apr 22, 2021
@eladkal eladkal requested a review from kaxil April 29, 2021 17:25
@kaxil kaxil added this to the Airflow 2.0.3 milestone Apr 29, 2021
ashb
ashb reviewed Apr 30, 2021
View reviewed changes
airflow/ti_deps/deps/trigger_rule_dep.py
ti.set_state(State.SKIPPED, session)
elif upstream_done and successes <= 0:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can successes be less than zero?

Copy link
Contributor Author

@r-richmond r-richmond Apr 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear to me; I'm just following the pattern I see below so that this is consistent.

airflow/airflow/ti_deps/deps/trigger_rule_dep.py

Lines 154 to 156 in 5e79b1e

if trigger_rule == TR.ONE_SUCCESS:
if successes <= 0:
yield self._failing_status(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair

@ashb ashb merged commit 32c6362 into apache:master May 4, 2021
@r-richmond r-richmond deleted the fix-one-success branch May 4, 2021 15:05
potiuk pushed a commit that referenced this pull request May 9, 2021
@r-richmond @potiuk
When one_success mark task as failed if no success (#15467)
9890cdc 
(cherry picked from commit 32c6362)
@ashb ashb modified the milestones: Airflow 2.0.3, Airflow 2.1 May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashb ashb ashb approved these changes

@kaxil kaxil Awaiting requested review from kaxil

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Airflow 2.1
Development

Successfully merging this pull request may close these issues.
3 participants
@r-richmond @kaxil @ashb