Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve AWS credential loading between S3 and DynamoDb code paths #2887

Merged
merged 4 commits into from
Sep 17, 2024

Conversation

rtyler
Copy link
Member

@rtyler rtyler commented Sep 16, 2024

This change implements a number of improvements to the code paths for loading credentials. This is a prerequisite to fixing assume role support #2879 but should also address a number of bugs I noticed:

  • Python libraries can pass in keys via storage_options which are used for configuration of the AmazonS3 ObjectStore, but those credentials would not be used in the construction of the DynamoDB connection
  • Using AWS credentials such as those from ~/.aws/profile or SSO would not be properly dropped into the AmazonS3 object store creation

There is some additional work that needs to come in to clean up how various options overrides are managed still.

Sponsored-by: Scribd Inc.

@rtyler rtyler added binding/python Issues for the Python package binding/rust Issues for the Rust crate storage/aws AWS S3 storage related labels Sep 16, 2024
@github-actions github-actions bot removed the binding/python Issues for the Python package label Sep 16, 2024
Copy link

codecov bot commented Sep 16, 2024

Codecov Report

Attention: Patch coverage is 74.48276% with 74 lines in your changes missing coverage. Please review.

Project coverage is 72.42%. Comparing base (546a344) to head (9846bac).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
crates/aws/src/credentials.rs 72.72% 28 Missing and 14 partials ⚠️
crates/aws/src/storage.rs 76.47% 31 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2887      +/-   ##
==========================================
- Coverage   72.61%   72.42%   -0.20%     
==========================================
  Files         131      131              
  Lines       40016    40011       -5     
  Branches    40016    40011       -5     
==========================================
- Hits        29057    28976      -81     
- Misses       9088     9166      +78     
+ Partials     1871     1869       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rtyler
Copy link
Member Author

rtyler commented Sep 16, 2024

Oh goodie, these tests fail in the GitHub Action runner environment and do not in local development. What a fun mystery this will be! 🕵️‍♂️

This change implements a number of improvements to the code paths for
loading credentials.  This is a prerequisite to fixing assume role
support delta-io#2879 but should also address a number of bugs I noticed:

* Python libraries can pass in keys via `storage_options` which are used
  for configuration of the AmazonS3 ObjectStore, but those credentials
  would not be used in the construction of the DynamoDB connection
* Using AWS credentials such as those from ~/.aws/profile or SSO would
  not be properly dropped into the AmazonS3 object store creation

There is some additional work that needs to come in to clean up how
various options overrides are managed still.

Sponsored-by: Scribd Inc.
the latest aws-config does a proper sequencing on the credential
resolution and avoids unnecessary queries to IMDSv2
@rtyler rtyler marked this pull request as ready for review September 17, 2024 01:34
rtyler added a commit to delta-io/kafka-delta-ingest that referenced this pull request Sep 17, 2024
rtyler added a commit to rtyler/kafka-delta-ingest that referenced this pull request Sep 17, 2024
@rtyler rtyler added this pull request to the merge queue Sep 17, 2024
Merged via the queue into delta-io:main with commit 83043b6 Sep 17, 2024
21 checks passed
@rtyler rtyler deleted the bug/iam-assume-role branch September 17, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate storage/aws AWS S3 storage related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants