More optimized lazy-loading of provider information #17304
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:CLI
area:dev-tools
area:webserver
potiuk
force-pushed
the
optimize-providers-manager-lazy-loading
branch
5 times, most recently
from
636dc8a to
6daa185
Compare
potiuk
changed the title
potiuk
force-pushed
the
optimize-providers-manager-lazy-loading
branch
from
6daa185 to
d2aa352
Compare
mik-laj
reviewed
Jul 29, 2021
ashb
reviewed
Jul 29, 2021
potiuk
force-pushed
the
optimize-providers-manager-lazy-loading
branch
2 times, most recently
from
6b95b4b to
5bce2c6
Compare
ashb
reviewed
Jul 29, 2021
With this change we truly lazy-load hooks and external_links only when we need them. Previously they were loaded when any of the properties of ProvidersManager was used, but with this change in some scenarios where only extra links are used or when we only need list of providers, but we do not need details on which custom hooks are needed, there will be much faster initialization. This is mainly for some CLI commands (for example `airlfow providers list` is much faster now), but also in some scenarios where for example .get_conn() is never used in Tasks, tasks might also never need to import/load the hooks and they might perform faster, with smaller memory footprint.
potiuk
force-pushed
the
optimize-providers-manager-lazy-loading
branch
from
5bce2c6 to
2cdd7e2
Compare
ashb
approved these changes
Jul 29, 2021
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
|
Tansient errors only - seems so local_task_job became flaky after recent "QUEUED" state intro - might be worth looking at @ephraimbuddy - I will make issues for those quickly (and qurarantine them) |
|
Should this be in 2.1.3 vs 2.2? |
|
Easy cherry-pick and it is rather save. I added it to 2.1.3 |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Aug 2, 2021
With this change we truly lazy-load hooks and external_links only when we need them. Previously they were loaded when any of the properties of ProvidersManager was used, but with this change in some scenarios where only extra links are used or when we only need list of providers, but we do not need details on which custom hooks are needed, there will be much faster initialization. This is mainly for some CLI commands (for example `airlfow providers list` is much faster now), but also in some scenarios where for example .get_conn() is never used in Tasks, tasks might also never need to import/load the hooks and they might perform faster, with smaller memory footprint. (cherry picked from commit 2dc7aa8)
kaxil
pushed a commit
that referenced
this pull request
Aug 17, 2021
With this change we truly lazy-load hooks and external_links only when we need them. Previously they were loaded when any of the properties of ProvidersManager was used, but with this change in some scenarios where only extra links are used or when we only need list of providers, but we do not need details on which custom hooks are needed, there will be much faster initialization. This is mainly for some CLI commands (for example `airlfow providers list` is much faster now), but also in some scenarios where for example .get_conn() is never used in Tasks, tasks might also never need to import/load the hooks and they might perform faster, with smaller memory footprint. (cherry picked from commit 2dc7aa8)
jhtimmins
pushed a commit
that referenced
this pull request
Aug 17, 2021
With this change we truly lazy-load hooks and external_links only when we need them. Previously they were loaded when any of the properties of ProvidersManager was used, but with this change in some scenarios where only extra links are used or when we only need list of providers, but we do not need details on which custom hooks are needed, there will be much faster initialization. This is mainly for some CLI commands (for example `airlfow providers list` is much faster now), but also in some scenarios where for example .get_conn() is never used in Tasks, tasks might also never need to import/load the hooks and they might perform faster, with smaller memory footprint. (cherry picked from commit 2dc7aa8)
Comment on lines
83
to
84
| # This is never executed, but tricks static analyzers (PyDev, PyCharm,) | ||
| # into knowing the types of these symbols, and what | ||
| # they contain. |
There was a problem hiding this comment.
@potiuk Was this line meant to be removed? (Sorry if I don't fully understand this PR, the comment just reads a bit funny.)
There was a problem hiding this comment.
Not at all - you can add PR to bring it back :).
There was a problem hiding this comment.
Merged! Quickest contribution EVER!
There was a problem hiding this comment.
Thanks! I did not expect my first contribution to be this quick 🙂
edwardwang888
added a commit
to edwardwang888/airflow
that referenced
this pull request
Aug 28, 2021
Fixes comment deleted in apache#17304
potiuk
pushed a commit
that referenced
this pull request
Aug 28, 2021
Fixes comment deleted in #17304
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this change we truly lazy-load hooks and external_links only
when we need them. Previously they were loaded when any of the
properties of ProvidersManager was used, but with this change
in some scenarios where only extra links are used or when we
only need list of providers, but we do not need details on
which custom hooks are needed, there will be much
faster initialization. This is mainly for some CLI commands
(for example
airlfow providers listis much faster now), butalso in some scenarios where for example .get_conn() is never
used in Tasks, tasks might also never need to import/load the hooks
and they might perform faster, with smaller memory footprint.