Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AssumeRoleProvider is not properly caching credentials #809

Closed
awsrick opened this issue May 19, 2023 · 2 comments
Closed

AssumeRoleProvider is not properly caching credentials #809

awsrick opened this issue May 19, 2023 · 2 comments
Assignees
Labels
bug This issue is a bug.

Comments

@awsrick
Copy link

awsrick commented May 19, 2023

Describe the bug

AssumeRoleProvider is not properly caching credentials when using as a credential provider for constructing an EC2 client. See below for the log messages and code snippets. This might be due to AssumeRoleProvider has its own inner cache because of the internal use of sts client.

Expected Behavior

AssumeRoleProvider should be using the same credential cache provided by sdk_config and should cache credentials after the first request.

Current Behavior

Seeing two cache miss messages in the logs every time when I make a request with a client constructed with AssumeRoleProvider. Seeing the messages with back to back requests.

{
    "ServiceName": "JobManagerExternalAPILambda",
    "caller": "event /workplace/user/BDJM/src/BirdDogExternalAPILambda/build/private/cargo-home/registry/src/-904edde59719864f/aws-credential-types-0.55.2/src/cache/lazy_caching.rs:113",
    "level": "info",
    "message": "credentials cache miss occurred; added new AWS credentials (took 10.6µs)",
    "requestId": "9c7155d1-f583-4406-a3bb-5e9f76554a24",
    "time": "2023-05-19T08:50:24.966896122Z",
    "xrayTraceId": "Root=1-64673850-8d1d61123b4a1d05f348cb54;Parent=6627a7a55146eea1;Sampled=0;Lineage=4300dddf:0"
}

    "ServiceName": "JobManagerExternalAPILambda",
    "caller": "event /workplace/user/BDJM/src/BirdDogExternalAPILambda/build/private/cargo-home/registry/src/-904edde59719864f/aws-credential-types-0.55.2/src/cache/lazy_caching.rs:113",
    "level": "info",
    "message": "credentials cache miss occurred; added new AWS credentials (took 33.45434ms)",
    "requestId": "9c7155d1-f583-4406-a3bb-5e9f76554a24",
    "time": "2023-05-19T08:50:25.000287289Z",
    "xrayTraceId": "Root=1-64673850-8d1d61123b4a1d05f348cb54;Parent=6627a7a55146eea1;Sampled=0;Lineage=4300dddf:0"
}

Reproduction Steps

Construct an EC2 client using AssumeRoleProvider like following and use EC2 client to make a request:

SDK config is only loaded once when application starts

  let sdk_config: SdkConfig = aws_config::from_env().credentials_cache(CredentialsCache::lazy()).load().await;
    let credentials_provider = {
        let provider = sdk_config
            .credentials_provider()
            .expect("No valid credentials found");

        let region = sdk_config.region().expect("No region found");

        AssumeRoleProvider::builder(role_arn)
            .session_name(role_session_name)
            .region(region.clone())
            .build(provider.clone())
    };
    
    let config_builder: Builder = Builder::from(sdk_config)
        .credentials_provider(credentials_provider);
    
    EC2Client::from_conf(config_builder.build())

Possible Solution

No response

Additional Information/Context

No response

Version

│   ├── aws-smithy-http v0.55.2
│   │   ├── aws-smithy-types v0.55.2
│   ├── aws-smithy-http-server v0.55.2
│   │   ├── aws-smithy-http v0.55.2 (*)
│   │   ├── aws-smithy-json v0.55.2
│   │   │   └── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-smithy-xml v0.55.2
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-smithy-async v0.55.2
│   ├── aws-smithy-client v0.55.2
│   │   ├── aws-smithy-async v0.55.2 (*)
│   │   ├── aws-smithy-http v0.55.2 (*)
│   │   ├── aws-smithy-http-tower v0.55.2
│   │   │   ├── aws-smithy-http v0.55.2 (*)
│   │   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-smithy-http v0.55.2 (*)
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
├── aws-arn v0.3.1
├── aws-config v0.55.2
│   ├── aws-credential-types v0.55.2
│   │   ├── aws-smithy-async v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-http v0.55.2
│   │   ├── aws-credential-types v0.55.2 (*)
│   │   ├── aws-smithy-http v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-types v0.55.2
│   │   │   ├── aws-credential-types v0.55.2 (*)
│   │   │   ├── aws-smithy-async v0.55.2 (*)
│   │   │   ├── aws-smithy-client v0.55.2 (*)
│   │   │   ├── aws-smithy-http v0.55.2 (*)
│   │   │   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-sdk-sso v0.27.0
│   │   ├── aws-credential-types v0.55.2 (*)
│   │   ├── aws-endpoint v0.55.2
│   │   │   ├── aws-smithy-http v0.55.2 (*)
│   │   │   ├── aws-smithy-types v0.55.2 (*)
│   │   │   ├── aws-types v0.55.2 (*)
│   │   ├── aws-http v0.55.2 (*)
│   │   ├── aws-sig-auth v0.55.2
│   │   │   ├── aws-credential-types v0.55.2 (*)
│   │   │   ├── aws-sigv4 v0.55.2
│   │   │   │   ├── aws-smithy-http v0.55.2 (*)
│   │   │   ├── aws-smithy-http v0.55.2 (*)
│   │   │   ├── aws-types v0.55.2 (*)
│   │   ├── aws-smithy-async v0.55.2 (*)
│   │   ├── aws-smithy-client v0.55.2 (*)
│   │   ├── aws-smithy-http v0.55.2 (*)
│   │   ├── aws-smithy-http-tower v0.55.2 (*)
│   │   ├── aws-smithy-json v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-types v0.55.2 (*)
│   ├── aws-sdk-sts v0.27.0
│   │   ├── aws-credential-types v0.55.2 (*)
│   │   ├── aws-endpoint v0.55.2 (*)
│   │   ├── aws-http v0.55.2 (*)
│   │   ├── aws-sig-auth v0.55.2 (*)
│   │   ├── aws-smithy-async v0.55.2 (*)
│   │   ├── aws-smithy-client v0.55.2 (*)
│   │   ├── aws-smithy-http v0.55.2 (*)
│   │   ├── aws-smithy-http-tower v0.55.2 (*)
│   │   ├── aws-smithy-json v0.55.2 (*)
│   │   ├── aws-smithy-query v0.55.2
│   │   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-smithy-types v0.55.2 (*)
│   │   ├── aws-smithy-xml v0.55.2 (*)
│   │   ├── aws-types v0.55.2 (*)
│   ├── aws-smithy-async v0.55.2 (*)
│   ├── aws-smithy-client v0.55.2 (*)
│   ├── aws-smithy-http v0.55.2 (*)
│   ├── aws-smithy-http-tower v0.55.2 (*)
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-types v0.55.2 (*)
├── aws-credential-types v0.55.2 (*)
├── aws-http v0.55.2 (*)
├── aws-sdk-dynamodb v0.27.0
│   ├── aws-credential-types v0.55.2 (*)
│   ├── aws-endpoint v0.55.2 (*)
│   ├── aws-http v0.55.2 (*)
│   ├── aws-sig-auth v0.55.2 (*)
│   ├── aws-smithy-async v0.55.2 (*)
│   ├── aws-smithy-client v0.55.2 (*)
│   ├── aws-smithy-http v0.55.2 (*)
│   ├── aws-smithy-http-tower v0.55.2 (*)
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-types v0.55.2 (*)
├── aws-sdk-ec2 v0.27.0
│   ├── aws-credential-types v0.55.2 (*)
│   ├── aws-endpoint v0.55.2 (*)
│   ├── aws-http v0.55.2 (*)
│   ├── aws-sig-auth v0.55.2 (*)
│   ├── aws-smithy-async v0.55.2 (*)
│   ├── aws-smithy-client v0.55.2 (*)
│   ├── aws-smithy-http v0.55.2 (*)
│   ├── aws-smithy-http-tower v0.55.2 (*)
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-query v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-smithy-xml v0.55.2 (*)
│   ├── aws-types v0.55.2 (*)
├── aws-sdk-iam v0.27.0
│   ├── aws-credential-types v0.55.2 (*)
│   ├── aws-endpoint v0.55.2 (*)
│   ├── aws-http v0.55.2 (*)
│   ├── aws-sig-auth v0.55.2 (*)
│   ├── aws-smithy-async v0.55.2 (*)
│   ├── aws-smithy-client v0.55.2 (*)
│   ├── aws-smithy-http v0.55.2 (*)
│   ├── aws-smithy-http-tower v0.55.2 (*)
│   ├── aws-smithy-json v0.55.2 (*)
│   ├── aws-smithy-query v0.55.2 (*)
│   ├── aws-smithy-types v0.55.2 (*)
│   ├── aws-smithy-xml v0.55.2 (*)
│   ├── aws-types v0.55.2 (*)
├── aws-sdk-sts v0.27.0 (*)
├── aws-sig-auth v0.55.2 (*)
├── aws-smithy-client v0.55.2 (*)
├── aws-smithy-http v0.55.2 (*)
├── aws-smithy-http-server v0.55.2 (*)
├── aws-smithy-http-tower v0.55.2 (*)
├── aws-types v0.55.2 (*)
│   ├── aws-sdk-dynamodb v0.27.0 (*)


### Environment details (OS name and version, etc.)

AWS Lambda

### Logs

_No response_
@awsrick awsrick added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 19, 2023
@ysaito1001 ysaito1001 self-assigned this May 19, 2023
@ysaito1001 ysaito1001 removed the needs-triage This issue or PR still needs to be triaged. label May 19, 2023
github-merge-queue bot pushed a commit to smithy-lang/smithy-rs that referenced this issue May 24, 2023
## Motivation and Context
Related to awslabs/aws-sdk-rust#809

## Description
It has been discovered that when `AssumeRoleProvider` is used, the Rust
SDK emits `credentials cache miss occurred` twice per request. The
reason why that log is shown twice is illustrated in the following
diagram:

![Screenshot 2023-05-19 at 4 10 20
PM](https://github.com/awslabs/smithy-rs/assets/15333866/c6cce018-c821-4b46-8d47-b414af7b4d1e)

One of the cache miss messages is due to the fact `AssumeRoleProvider`
internally uses an STS client, which, in turn, is wrapped by a
`LazyCredentialsCache` by default. However, that use of
`LazyCredentialsCache` is pointless because caching is already in effect
with the outermost `LazyCredentialsCache`.

This PR adds a new kind of `CredentialsCache`, `NoCredentialsCache`. As
its name suggests, it simplify delegates `provide_cached_credentials` to
the underlying provider's `provide_credentials` with no caching
functionality. We then update `SsoCredentialsProvider`,
`AssumeRoleProvider`, and `WebIdentityTokenCredentialsProvider` to use
`NoCredentialsCache` for their STS clients so the logs won't show
`credentials cache miss occurred` twice per request.

## Testing
- Added unit tests for `NoCredentialsCache`
- Updated unit test for `AssumeRoleProvider` to verify
`NoCredentialsCache` is used by default

## Checklist
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: Yuki Saito <[email protected]>
hlbarber pushed a commit to smithy-lang/smithy-rs that referenced this issue Jun 20, 2023
Related to awslabs/aws-sdk-rust#809
It has been discovered that when `AssumeRoleProvider` is used, the Rust
SDK emits `credentials cache miss occurred` twice per request. The
reason why that log is shown twice is illustrated in the following
diagram:

![Screenshot 2023-05-19 at 4 10 20
PM](https://github.com/awslabs/smithy-rs/assets/15333866/c6cce018-c821-4b46-8d47-b414af7b4d1e)

One of the cache miss messages is due to the fact `AssumeRoleProvider`
internally uses an STS client, which, in turn, is wrapped by a
`LazyCredentialsCache` by default. However, that use of
`LazyCredentialsCache` is pointless because caching is already in effect
with the outermost `LazyCredentialsCache`.

This PR adds a new kind of `CredentialsCache`, `NoCredentialsCache`. As
its name suggests, it simplify delegates `provide_cached_credentials` to
the underlying provider's `provide_credentials` with no caching
functionality. We then update `SsoCredentialsProvider`,
`AssumeRoleProvider`, and `WebIdentityTokenCredentialsProvider` to use
`NoCredentialsCache` for their STS clients so the logs won't show
`credentials cache miss occurred` twice per request.
- Added unit tests for `NoCredentialsCache`
- Updated unit test for `AssumeRoleProvider` to verify
`NoCredentialsCache` is used by default
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: Yuki Saito <[email protected]>
aws-sdk-rust-ci pushed a commit that referenced this issue Aug 3, 2023
…#2720)

## Motivation and Context
Related to #809

## Description
It has been discovered that when `AssumeRoleProvider` is used, the Rust
SDK emits `credentials cache miss occurred` twice per request. The
reason why that log is shown twice is illustrated in the following
diagram:

![Screenshot 2023-05-19 at 4 10 20
PM](https://github.com/awslabs/smithy-rs/assets/15333866/c6cce018-c821-4b46-8d47-b414af7b4d1e)

One of the cache miss messages is due to the fact `AssumeRoleProvider`
internally uses an STS client, which, in turn, is wrapped by a
`LazyCredentialsCache` by default. However, that use of
`LazyCredentialsCache` is pointless because caching is already in effect
with the outermost `LazyCredentialsCache`.

This PR adds a new kind of `CredentialsCache`, `NoCredentialsCache`. As
its name suggests, it simplify delegates `provide_cached_credentials` to
the underlying provider's `provide_credentials` with no caching
functionality. We then update `SsoCredentialsProvider`,
`AssumeRoleProvider`, and `WebIdentityTokenCredentialsProvider` to use
`NoCredentialsCache` for their STS clients so the logs won't show
`credentials cache miss occurred` twice per request.

## Testing
- Added unit tests for `NoCredentialsCache`
- Updated unit test for `AssumeRoleProvider` to verify
`NoCredentialsCache` is used by default

## Checklist
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: Yuki Saito <[email protected]>
@ysaito1001
Copy link
Collaborator

smithy-lang/smithy-rs#2720 has replaced the inner cache of AssumeRoleProvider by NoCredentialsCache, removing duplicated cache misses in the log message.

@github-actions
Copy link

github-actions bot commented Aug 4, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

2 participants