IAM Policy attached to EC2 not working when ran from inside Docker container

[user9747]

Describe the bug

The EC2 instance i use has a IAM Role attached to it which allows GetObject from s3 bucket.

The above code work if run it on the host EC2, but doesn't work if run it in a docker. Also i found that it works if set the docker to run with network as host.

Expected Behavior

Attached iam role not picked up by AWS sdk inside docker

Current Behavior

For some reason imds is not working from inside docker.

Reproduction Steps

    let config = aws_config::from_env()
        .region(aws_sdk_s3::Region::new("ap-south-1"))
        .load()
        .await;
    let client = Client::new(&config);
let resp = client
        .get_object()
        .bucket(bucket_name)
        .key(key)
        .send()
        .await?;

Possible Solution

No response

Additional Information/Context

No response

Version

aws-config = "0.12.0" aws-sdk-s3 = "0.12.0"

Environment details (OS name and version, etc.)

ubuntu

Logs

[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation;
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; operation="GetObject"
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; service="s3"
[2022-05-22T19:44:03Z INFO aws_config::default_provider::credentials] provide_credentials; provider=default_chain
[2022-05-22T19:44:03Z INFO aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=Environment context=environment variable not set
[2022-05-22T19:44:03Z INFO aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=Profile context=No profiles were defined
[2022-05-22T19:44:03Z INFO aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
[2022-05-22T19:44:03Z INFO aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation;
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; operation="get"
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; service="imds"
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation;
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; operation="get-token"
[2022-05-22T19:44:03Z INFO aws_smithy_http_tower::parse_response] send_operation; service="imds"
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; status="dispatch_failure"
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; message=timeout: HTTP read timeout occurred after 1s
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; status="construction_failure"
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; message=Failed to load session token: timeout: HTTP read timeout occurred after 1s
[2022-05-22T19:44:04Z INFO aws_config::meta::credentials::chain] provider in chain did not provide credentials provider=Ec2InstanceMetadata context=could not communicate with imds: timeout: HTTP read timeout occurred after 1s
[2022-05-22T19:44:04Z INFO aws_http::auth] provider returned CredentialsNotLoaded, ignoring
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; status="construction_failure"
[2022-05-22T19:44:04Z INFO aws_smithy_http_tower::parse_response] send_operation; message=No credentials in the property bag

[jmklix]

It looks like you are running into a timeout message=timeout: HTTP read timeout occurred after 1s
It might be something with how you have your docker image setup. Can you try increasing the timeout to see if that fixes it?

[catalinw]

Hello! We had the exact same problem and we managed to find a workaround (increasing the timeout as suggested above didn't work):

How we figured out what was the problem:

There was this similar problem with aws go sdk in the past:
aws/aws-sdk-go#2972
As you can see for this issue, the workaround was to increase the http put response hop limit to 2 in the ec instance.
This makes sense also for our problem because it says in the aws docs (here) that:

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop.

So it means that, running the rust code into docker, instead of machine -> imds, it's going container -> machine -> imds, so 2 hops instead of one.

We added 2 new extra docker machine settings and works fine now:

amazonec2-metadata-token=required
amazonec2-metadata-token-response-hop-limit=2

[jmklix]

Thanks cataliw for that workaround. Does this work for you user9747?

[zhrebicek]

FYI

I have reproduced the issue and fixed by those 2 new extra docker machine settings:

amazonec2-metadata-token=required
amazonec2-metadata-token-response-hop-limit=2

Is it worth adding somewhere to docs?

[rcoh]

closing since this isn't a bug—a doc PR is open

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Zero-Fanker]

Hello! We had the exact same problem and we managed to find a workaround (increasing the timeout as suggested above didn't work):

How we figured out what was the problem:

There was this similar problem with aws go sdk in the past: aws/aws-sdk-go#2972 As you can see for this issue, the workaround was to increase the http put response hop limit to 2 in the ec instance. This makes sense also for our problem because it says in the aws docs (here) that:

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop.

So it means that, running the rust code into docker, instead of machine -> imds, it's going container -> machine -> imds, so 2 hops instead of one.

We added 2 new extra docker machine settings and works fine now:

amazonec2-metadata-token=required
amazonec2-metadata-token-response-hop-limit=2

hi catalinw, i referred to AWS doc and found no such amazonec2-metadata-token=required amazonec2-metadata-token-response-hop-limit=2 options but only

aws ec2 modify-instance-metadata-options \
    --instance-id {inst_id} \
    --http-tokens required \
    --http-endpoint enabled \
    --http-put-response-hop-limit 2

Will it work the same way or could you provide an example to show how and where to configure such parameters ?

[Fuuzetsu]

For anyone else that stumbles upon this:

• AWS cli (both v1 and v2) support IMDSv1 and IMDSv2: it'll try v2 first, quickly giving up and using v1 if that fails
• rusoto only supports IMDSv1 (IMDSv2 rusoto/rusoto#1818) which works for now but rusoto is out of date and this won't work if you make IMDSv2 required on your instances as is now recommended
• aws-sdk-rust only supports IMDSv2 and there's no plan to support IMDSv1 ever: Support IMDSv1 for authentication #520

Therefore the increase in hop limit is not really a workaround, it is the solution. I thought I'd just make the situation clear in case someone else wastes a bunch of time trying to figure out why their existing code works but it stops working when aws-sdk-rust is used instead.