Update AWS Connection docs and deprecate some extras #24670
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
kind:documentation
provider:amazon-aws
o-nikolas
requested changes
Jun 27, 2022
| @@ -83,6 +83,15 @@ def create_session(self) -> boto3.session.Session: | |||
| self.extra_config["session_kwargs"], | |||
| ) | |||
| session_kwargs = self.extra_config["session_kwargs"] | |||
|
|
|||
| if "profile" in self.extra_config and "s3_config_file" not in self.extra_config: | |||
There was a problem hiding this comment.
Why is s3 specific logic being added to the AWS base session class?
There was a problem hiding this comment.
Why initially it added in AWS Hook or why I add this warning?
Answer for fist part: #2532 (comment)
Answer for second part: This parameter exist in documentation since version 1.1.0 without mentioning how it actually works, I've just add warning to inform user.
|
Fix grammar errors and deprecate config from "local credentials file" Updated docs
|
potiuk
approved these changes
Jul 4, 2022
95 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
aws_account_id,aws_iam_role,external_id,s3_config_file,s3_config_format,profileGood candidate for future deprecation:
s3_config_file,s3_config_format,profileThis options migrated from legacy S3Hook 3865836 in Airflow 1.9, and never been documented and well-tested. This use for parse local credentials file in selected formats:boto config files - boto not maintained for many yearsAWS Shared Credentials file. In current implementation only get parts credentials. If need to specify non default location to AWS Shared Credentials and AWS Config files better to use configuration of botocore.sessionSession. In this case also could use additional abilities such as credential_process, and profiles inheritance.s3cmd - Seems like this utility never been part of Airflowhostmight confuse users (actually couple of issues exists in Github) better rename toendpoint_urlsession_kwargs- onlyprofile_namecould be use as part of this dictionary, other boto3.session.Session arguments stored directly in extra@ferruzzi @o-nikolas