-
Notifications
You must be signed in to change notification settings - Fork 406
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: improve AWS credential loading between S3 and DynamoDb code paths #2887
Conversation
b2bad8a
to
1cd8d98
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2887 +/- ##
==========================================
- Coverage 72.61% 72.42% -0.20%
==========================================
Files 131 131
Lines 40016 40011 -5
Branches 40016 40011 -5
==========================================
- Hits 29057 28976 -81
- Misses 9088 9166 +78
+ Partials 1871 1869 -2 ☔ View full report in Codecov by Sentry. |
Oh goodie, these tests fail in the GitHub Action runner environment and do not in local development. What a fun mystery this will be! 🕵️♂️ |
a5a9704
to
86ad7c2
Compare
This change implements a number of improvements to the code paths for loading credentials. This is a prerequisite to fixing assume role support delta-io#2879 but should also address a number of bugs I noticed: * Python libraries can pass in keys via `storage_options` which are used for configuration of the AmazonS3 ObjectStore, but those credentials would not be used in the construction of the DynamoDB connection * Using AWS credentials such as those from ~/.aws/profile or SSO would not be properly dropped into the AmazonS3 object store creation There is some additional work that needs to come in to clean up how various options overrides are managed still. Sponsored-by: Scribd Inc.
the latest aws-config does a proper sequencing on the credential resolution and avoids unnecessary queries to IMDSv2
86ad7c2
to
9e2d000
Compare
…ssume role support See delta-io/delta-rs#2887
…ssume role support See delta-io/delta-rs#2887
This change implements a number of improvements to the code paths for loading credentials. This is a prerequisite to fixing assume role support #2879 but should also address a number of bugs I noticed:
storage_options
which are used for configuration of the AmazonS3 ObjectStore, but those credentials would not be used in the construction of the DynamoDB connectionThere is some additional work that needs to come in to clean up how various options overrides are managed still.
Sponsored-by: Scribd Inc.