-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deprecate S3PrefixSensor
and S3KeySizeSensor
in favor of S3KeySensor
#22737
Conversation
Update S3KeySensor to handle multiple files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Logic seems OK
left few comments
Co-authored-by: Ash Berlin-Taylor <[email protected]>
Address feedbacks from PR apache#22737
While working on addressing feedbacks from this PR I realized that |
I only left tests related to deprecation warnings and default function in |
I also edited the PR description to add a section to explain the purpose of |
S3PrefixSensor
and S3KeySizeSensor
in favor of S3KeySensor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job @vincbeck
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Deprecate
S3PrefixSensor
and useS3KeySensor
instead.Why deprecating
S3PrefixSensor
?S3PrefixSensor
andS3KeySensor
are doing pretty much the same thing so to avoid duplicates, we should deprecate one.Also
S3PrefixSensor
does not behave the way it is described by the documentation in comments and the way you expect by the name.S3PrefixSensor
does not wait for a given prefix in S3 to exist but it waits for a given folder in S3 to exist (given the delimiter is/
). Here are some examples I ran for testing.prefix="test"
.true
when directorytest
existsprefix="test"
.false
when directorytest
does not exist and filetest
existsprefix="test"
.false
when directorytest
does not exist and filetest2
existsprefix="tes"
.false
when directorytest
existsThis misalignment between expected behavior and actual one is confusing for users. Example of thread where a user does not understand why
S3PrefixSensor
behave this way.Why updating
S3KeySensor
?In order to be backward compatible we want to use
S3KeySensor
instead.S3PrefixSensor
accept a list of files as input, soS3KeySensor
should be tooWhat is
default_check_fn
and why do we need it?S3KeySizeSensor
works this way: you can provide a custom function to this sensor to apply an additional check layer. If you dont provide one,default_check_fn
is used instead. We will need it to be backward compatible and keep the current behavior ofS3KeySizeSensor
as it is today.