Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix literal cross product expansion #23434

Merged
merged 2 commits into from
May 4, 2022

Conversation

jedcunningham
Copy link
Member

Closes: #23425

parse_time_mapped_ti_count wasn't taking into account the cross product behavior of expansion.

@jedcunningham jedcunningham added this to the Airflow 2.3.1 milestone May 3, 2022
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label May 3, 2022
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh!

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label May 3, 2022
@github-actions
Copy link

github-actions bot commented May 3, 2022

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Comment on lines +798 to +801
if total == 0:
total = len(value)
else:
total *= len(value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could have total = 1 at the beginning of the function instead…?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that, and that's how I found the .expand() bug. If we can assume we will always have at least something with a length, we could, but this felt more defensive.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s do this right now and maybe change it when we fix the expand bug. That way we can keep main working correctly.

@jedcunningham jedcunningham force-pushed the fix_cross_product_literal branch from 6c1b1df to 8946e36 Compare May 3, 2022 17:47
@jedcunningham jedcunningham force-pushed the fix_cross_product_literal branch from 8946e36 to 3be09f4 Compare May 4, 2022 04:45
@jedcunningham jedcunningham reopened this May 4, 2022
@jedcunningham jedcunningham merged commit 3fb8e0b into apache:main May 4, 2022
@jedcunningham jedcunningham deleted the fix_cross_product_literal branch May 4, 2022 19:02
ephraimbuddy pushed a commit that referenced this pull request May 8, 2022
ephraimbuddy pushed a commit that referenced this pull request May 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mapping over multiple parameters results in 1 task fewer than expected
4 participants