-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert sftp hook to use paramiko instead of pysftp #24512
Convert sftp hook to use paramiko instead of pysftp #24512
Conversation
d1f1ff5
to
c98f39d
Compare
One other note I forgot to include: the default for The primary motivation for doing this was to ensure that the SFTP connections can support all of the parameters supported by |
9d424f3
to
df9c8eb
Compare
df9c8eb
to
2262400
Compare
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
@@ -214,7 +201,18 @@ def create_directory(self, path: str, mode: int = 777) -> None: | |||
:param mode: int representation of octal mode for directory | |||
""" | |||
conn = self.get_conn() | |||
conn.makedirs(path, mode) | |||
if self.isdir(path): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice an detailed errors.
This PR is a major change to the SFTP provider to remove the dependency on
pysftp
and useparamiko
via theSSHHook
instead. The provider was previously inconsistent in its use ofpysftp
vs theSSHHook
. TheSFTPHook
usedpysftp
to actually create its connection and perform operations, but it inherited fromSSHHook
. TheSFTPOperator
did not use theSFTPHook
orpysftp
and instead used theSSHHook
to create its connection and use the SFTP operations provided byparamiko
. This PR rewrites the provider so theSFTPHook
uses theSSHHook
andparamiko
(viaSSHHook
) and theSFTPOperator
uses theSFTPHook
. This enables support for all of the parameters supported by theSSHHook
and creates consistency within the SFTP provider. There should be no breaking changes aside from the removal of previously deprecated parameters in theSFTPHook
sincepysftp
usedparamiko
under the hood anyway.A number of changes are contained in this PR:
ciphers
as an optional extra parameter, which is used by theSFTPHook
pysftp
that was used by theSFTPHook
has been incorporated directly into the providerSFTPHook
supports passing anSSHHook
to continue supporting that option in theSFTPOperator
; however, this is marked as deprecated and should be removed in a future major version.SFTPOperator
warns users to pass anSFTPHook
instead now._make_intermediate_dirs
was removed fromSFTPOperator
since thecreate_directory
method onSFTPHook
supports creating parent directories of subdirectories (functionally equivalent)SFTPOperator
Tests have been written to achieve 100% code coverage and all tests are passing. Previously existing tests were only modified where required to ensure backward compatibility with the provider's original functionality. Please double-check my tests though as I'm not confident in my test writing abilities yet. Docs have been updated as well.
I will note there was brief discussion with @malthe on Slack to consider using
parallel-ssh
instead but I opted to hold off on that for now. That could be a good future update since some have reported issues withparamiko
(for example, #16286).