Remove executor_class from Job - fixing backfil for custom executors #32219
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
the
area:Scheduler
potiuk
requested review from
ephraimbuddy,
dstandish,
o-nikolas and
jedcunningham
The executor_class in job was used in the past to pass information about executor being used but it has been replaced by executor passed via JobRunner when needed. The executor_class has only been used in backfill job, but it has been used wrongly - because it had no fully qualified path there. The right way of using it is to check the executor passed via JobRunner, and since executor_class is not used any more, we can safely remove it. This fixes a bug when custom exceutors could not be used during backfill. Co-Authored-By: Andrew Halpern <[email protected]>
potiuk
force-pushed
the
fix-executor-class-check-for-backfill
branch
from
99ebdc9 to
7783a3f
Compare
potiuk
changed the title
dstandish
approved these changes
Jun 27, 2023
|
More complete version of #32101 proposed by @adh-wonolo (thanks for the fix propal and being co-author of this change :) ) |
uranusjr
approved these changes
Jun 28, 2023
o-nikolas
approved these changes
Jun 28, 2023
ephraimbuddy
pushed a commit
that referenced
this pull request
Jul 6, 2023
…32219) The executor_class in job was used in the past to pass information about executor being used but it has been replaced by executor passed via JobRunner when needed. The executor_class has only been used in backfill job, but it has been used wrongly - because it had no fully qualified path there. The right way of using it is to check the executor passed via JobRunner, and since executor_class is not used any more, we can safely remove it. This fixes a bug when custom exceutors could not be used during backfill. Co-authored-by: Andrew Halpern <[email protected]> (cherry picked from commit df4026a)
53 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The executor_class in job was used in the past to pass information about executor being used but it has been replaced by executor passed via JobRunner when needed. The executor_class has only been used in backfill job, but it has been used wrongly - because it had no fully qualified path there.
The right way of using it is to check the executor passed via JobRunner, and since executor_class is not used any more, we can safely remove it
This fixes a bug when custom executors could not be used
during backfill.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.