-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for EKS 'IAM for service accounts' not default [1.11.623, 1.11.653] #2136
Comments
Something like this works:
So it's just a problem that |
Hi @thundergolfer Lines 45 to 50 in 5bafd0d
Could it be possible that you had some other credentials providers available on your credential chain and took precedence? |
Hey @zoewangg thanks for the reply, Yeh you're right seems it landed in the default chain in Given that someone else has seen this is, I'm confident something is going on here. Do you have any advice on how I would best track this down? Perhaps simply enabling |
Yeah, it'd be great if you could enable debug logging and provide the logs here (Be sure to redact any sensitive info from the logs) |
I am debugging a similar issue. AWS SDK fails to recognize Debug logs captured below,
|
In those logs What's that on the end here -> |
I think I'm experiencing the same issue or something similar. Yesterday I updated my application to use a service account and also updated the aws-java-sdk to v1.11.673. Adding my logs here in case they help. The app fails to deploy to EKS and hits CrashLoopBackOff. In the logs I can't see WebIdentityTokenCredentialsProvider - {"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@54e7391d: Unable to load Client Side Monitoring configurations from environment variables!","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@50b8ae8d: Unable to load Client Side Monitoring configurations from system properties variables!","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@3c8bdd5b: Unable to load config file","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.479+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.479+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
{"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000}
... this repeats many times I also get a StackOverflowError with the stack trace after the
|
When we implemented IRSA for our Java apps, we found that as configured in the default chain instance profile creds outrank WebIdentity; whereas in the Ruby and Go SDKs WebIdentity took precedence. Since the kubelet auths with IAM there's no way to not have an instance profile, and we don't have the option of blocking the metadata API for pods until every last service in our cluster is using IRSA. Unless the Java SDK changes its precedence order to match the other SDKs we're going to need to provide our own chains in all of our Java apps, which is less than ideal. |
I was experiencing a StackOverflowError becuase another library was pulling in an older version of aws-java-sdk-sts which used to initialise a default credentials provider internally, updating this resolved the StackOverflowError. Once that issue was resolved, as mentioned above we had to provide a credential chain to give WebIdentityTokenCredentialsProvider precedence. It would be helpful if the java sdk's default credentials provider precedence order was changed to give the WebIdentityTokenCredentialsProvider higher precedence. |
I also encounter same issue and I upgraded my aws-java-sdk-sts lib version(1.11.623) to same as s3 lib version (1.11.623),it fixed. |
In our case all our SDK libraries are on |
aws/aws-sdk-java-v2#1583 will increase the priority of the web identity environment variables in 2.x (and a similar change will be coming out for 1.11.x) to be higher than the profile file properties, but this is the only issue we could reproduce on our end. Please make sure you're using the latest SDK runtime/core as well as the latest client version. |
We've fixed this by adding |
The fix has been released as part of https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md#111704-2020-01-09 |
We added |
@thundergolfer Thanks for reporting back! Closing the issue. Feel free to reopen if you have further question. |
So to clarify, we need to upgrade SDKs to |
I tried to upgrade
Changing my client initialization to this, it works:
If Detail: I'm also using IAM Policy attached on EC2 worker nodes (EKS cluster purposes). |
@jhonatanmorais before investigating it further, can you make sure there's no conflicting versions in your environment? Could you run |
|
@jhonatanmorais so I assume you have tried |
@bqnguyen94 I've tried only those two scenarios mentioned above. |
this is required to support EKS IAM roles for service accounts, see: aws/aws-sdk-java#2136 Co-authored-by: Valentino Gattuso <[email protected]>
Despite since `software.amazon.awssdk:2.10.11`, the `WebIdentityTokenFileCredentialsProvider` is part of the default credentials chain, it will only be enabled if `software.amazon.awssdk:sts` is on the classpath, which is mentioned in the javadocs: "Use of this credentials provider requires the 'sts' module to be on the classpath". See aws/aws-sdk-java#2136
Despite since `software.amazon.awssdk:2.10.11`, the `WebIdentityTokenFileCredentialsProvider` is part of the default credentials chain, it will only be enabled if `software.amazon.awssdk:sts` is on the classpath, which is mentioned in the javadocs: "Use of this credentials provider requires the 'sts' module to be on the classpath". See aws/aws-sdk-java#2136
Despite since `software.amazon.awssdk:2.10.11`, the `WebIdentityTokenFileCredentialsProvider` is part of the default credentials chain, it will only be enabled if `software.amazon.awssdk:sts` is on the classpath, which is mentioned in the javadocs: "Use of this credentials provider requires the 'sts' module to be on the classpath". See aws/aws-sdk-java#2136
Despite since `software.amazon.awssdk:2.10.11`, the `WebIdentityTokenFileCredentialsProvider` is part of the default credentials chain, it will only be enabled if `software.amazon.awssdk:sts` is on the classpath, which is mentioned in the javadocs: "Use of this credentials provider requires the 'sts' module to be on the classpath". See aws/aws-sdk-java#2136
Description
Despite the documentation saying a minimum version of
1.11.623
should allow IAM Service Accounts to works, I'm running pods and seeing that that run with the IAM role of the node and not the role linked to the IAM ServiceAccount I am running as.Another using describes the same problem with
1.11.653
here.I have discovered a hack that fixes the issue:
That above identity request returns the correct role, and then subsequently doing something like this,
AmazonSNSAsyncClientBuilder.defaultClient()
, will get me a client authenticated to the IAM role associated with the service account.The text was updated successfully, but these errors were encountered: