Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add processor_subdir to import_error table to handle multiple dag processors #35956

Merged
merged 2 commits into from
Nov 30, 2023

Conversation

tirkarthi
Copy link
Contributor

Add processor_subdir to import_error table to update the value while creating ImportError entry. Use the processor_subdir to filter for ImportError entries that belong to the current dag processor.

closes: #35949
related: #35949

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @mhenc ? Maybe you would like to review/comment ?

@potiuk potiuk requested a review from mhenc November 30, 2023 13:14
@potiuk potiuk added this to the Airflow 2.8.0 milestone Nov 30, 2023
Copy link
Contributor

@mhenc mhenc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thank you for the fix!

@potiuk potiuk merged commit 1a3eeab into apache:main Nov 30, 2023
47 checks passed
@tirkarthi
Copy link
Contributor Author

The other issue we had on switching from one dag processor to multiple ones using subdir was that old import errors were not deleted as they had different directory. It will be the same issue on switching to single dag processor from subdir based approach and needs a one time cleanup of import error table. Just noting this down if anyone encounters it.

Thanks @mhenc and @potiuk for the review .

@ephraimbuddy ephraimbuddy added the type:improvement Changelog: Improvements label Dec 5, 2023
ephraimbuddy pushed a commit that referenced this pull request Dec 5, 2023
…cessors (#35956)

* Add processor_subdir to import_error table to handle multiple dag processors.

* Fix migration table name.

(cherry picked from commit 1a3eeab)
@seanmuth
Copy link

seanmuth commented Feb 8, 2024

Took me a while to find this PR and specifically your comment above @tirkarthi , but thank you for noting this behavior! Made for an easy fix and understanding of how this occurred on a client's airflow deployment 🖖

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler kind:documentation type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dag processor deletes import errors of other dag processors thinking the files don't exist
5 participants