-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix mypy providers #20190
Merged
Merged
Fix mypy providers #20190
Changes from 32 commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
a1c532e
[16185] Added LocalKubernetesExecutor to breeze supported executors
subkanthi 6cbb506
Merge branch 'master' of github.com:subkanthi/airflow
subkanthi d16a343
Revert "[16185] Added LocalKubernetesExecutor to breeze supported exe…
subkanthi a336e17
Merge branch 'apache:main' into master
subkanthi 382eedf
Merge branch 'apache:main' into master
subkanthi c3030fb
Merge branch 'apache:main' into master
subkanthi 1c3c029
Merge branch 'apache:main' into master
subkanthi 0eb6598
Merge branch 'apache:main' into master
subkanthi 8ef0bcc
[19891] Fixed mypy in spark_sql
subkanthi c719166
Fixed mypy error in jira hook
subkanthi a503ef5
Fixed mypy in cassandra example_dag
subkanthi 37e2fc6
Fixed mypy errors in asana hooks
subkanthi 3c976c5
Fixed mypy errors in telegram provider
subkanthi 3638cd0
Fixed mypy errors in providers/trino
subkanthi f8cba96
Removed adding arguments to cassandra example dag
subkanthi 31ad8db
Removed default arguments in example_asana
subkanthi 6a117e4
Fixed mypy errors in postgres and slack
subkanthi 35a344e
Merge branch 'apache:main' into master
subkanthi 0f60ae4
Fixed static checks in trino
subkanthi 56b414c
Merge branch 'master' of github.com:subkanthi/airflow into fix_mypy_p…
subkanthi 8d47486
Merge branch 'apache:main' into master
subkanthi b4daedc
Merge branch 'master' of github.com:subkanthi/airflow into fix_mypy_p…
subkanthi 7a31cd1
Fixed unit tests in test_telegram
subkanthi e21a9a8
Removed unnecessary files
subkanthi 1fa154a
Fixed mypy errors in providers/docker
subkanthi a997e30
Merge branch 'fix_mypy_providers' of github.com:subkanthi/airflow int…
subkanthi 12c628e
Addressed PR review comments
subkanthi dcf537d
More mypy fixes
subkanthi f450679
Merge branch 'apache:main' into master
subkanthi 96b7a9b
Fixed merge conflicts
subkanthi 67c6f5e
Merge branch 'apache:main' into master
subkanthi dbfdcc5
Resolved conflicts and static checks
subkanthi a09d628
Merge branch 'apache:main' into master
subkanthi 816b976
Merge branch 'master' of github.com:subkanthi/airflow into fix_mypy_p…
subkanthi fa68b40
Removed type:ignore in Docker decorator
subkanthi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the stringified typing can safely be removed then the
TYPE_CHECKING
statement can also be removed. You'll need to confirm though with some testing. MyPy should complain if it's an issue.@kaxil @potiuk Another example of
get_connection()
in a hook's__init__()
here. Oddly enough, this hook has aget_conn()
method but it's a null method.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don’t think this is going to work at runtime. The stringified version is still needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm @josh-fell (and @kaxil ) - Something struck, me I just looked closer at the #18339
I think while discussing it I made a silent assumption (which I see now was wrong ) that the connection was created as part of "operator's" init but this is about creating it in the Hook's init, which IMHO is quite legitimate use case (as long as you do not instantiate the Hook in the operator's init(). And it is pretty common pattern in Airflow (and one we actually encourage).
I double-checked and I looked at the Databricks code and the only place I could see it being instantiated was _get_hook() and the only place where _get_hook() is called was inside "execute" and "on_kill" method of the Databricks operator - so that all sounds pretty legitimate.
Was not the whole issue caused by a misunderstanding of who's init it was? I think we have maaaaany cases where Hook is created "on-demand" in execute() method of the operator, also if you decide to create the Hook() inside the
@task
-decorated functions, it should work really well.Are my eyes cheating me ? O r maybe I miss something?