CLI v2 ecs execute-command does not support SSO Session profiles

[cupakromer]

Describe the bug

ecs execute-command does not support SSO session based profiles. Attempting to use the command results in either a missing configuration error or a missing cache file error.

Missing Configuration Error

SessionId: ecs-execute-command-0123456789 : ----------ERROR------- Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: error while creating new KMS service, Error creating new aws sdk session profile "my-sso-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

Missing Cache File Error

SessionId: ecs-execute-command-0123456789 : ----------ERROR------- Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: Error calling KMS GenerateDataKey API: SSOProviderInvalidToken: the SSO session has expired or is invalid caused by: open /Users/me/.aws/sso/cache/a00fce5cb007c23a469c136160398db65edcb180.json: no such file or directory

Expected Behavior

Running the aws ecs execute-command should behave like other CLI commands and work with non-legacy SSO profiles.

$ aws ecs execute-command \
  --profile my-sso-profile \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --interactive \
  --task arn:aws:ecs:us-east-1:redacted:task/my-ecs-cluster/redacted \
  --command 'sh'

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0123456789
This session is encrypted using AWS KMS.
# 

Current Behavior

Many AWS CLI commands work just fine with the SSO session based profiles:

$ aws s3 ls --profile my-sso-profile
2023-01-27 11:27:11 myfirst-bucket
2022-11-01 11:42:57 legacy-bucket

$ aws ecs list-tasks \
  --desired-status RUNNING \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --profile my-sso-profile
{
    "taskArns": [
        "arn:aws:ecs:us-east-1: redacted:task/my-ecs-cluster/redacted",
        "arn:aws:ecs:us-east-1: redacted:task/my-ecs-cluster/redacted2",
        "arn:aws:ecs:us-east-1: redacted:task/my-ecs-cluster/redacted3"
    ]
}

Attempting to use the same profile with ecs execute-command fails:

$ aws ecs execute-command \
  --profile my-sso-profile \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --interactive \                                                                                  
  --task arn:aws:ecs:us-east-1:redacted:task/my-ecs-cluster/redacted \
  --command 'sh'

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0123456789


SessionId: ecs-execute-command-0123456789 : 
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error:
Failed to process action KMSEncryption: error while creating new KMS service, Error creating new aws sdk
session profile "my-sso-profile" is configured to use SSO but is missing required configuration:
sso_region, sso_start_url

If the listed missing configuration parameters are added, then the error changes to:

Starting session with SessionId: ecs-execute-command-0123456789


SessionId: ecs-execute-command-0123456789 : 
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error:
Failed to process action KMSEncryption: Error calling KMS GenerateDataKey API: SSOProviderInvalidToken:
the SSO session has expired or is invalid caused by:
open /Users/me/.aws/sso/cache/a00fce5cb007c23a469c136160398db65edcb180.json: no such file or directory

Switching to a previously configured legacy version of the profile allows ecs execute-command to run successfully:

$ aws ecs execute-command --profile my-legacy-profile ...

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0123456789
This session is encrypted using AWS KMS.
#

Reproduction Steps

Start with the following configuration:

# ~/.aws/config
[profile my-legacy-profile]
sso_start_url = https://my-sso.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789000
sso_role_name = DevAccess
region = us-east-1
output = json
[profile my-sso-profile]
sso_session = my-sso-session
sso_account_id = 123456789000
sso_role_name = DevAccess
region = us-east-1
output = json
[sso-session my-sso-session]
sso_start_url = https://my-sso.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Login to the non-legacy SSO profile. Then using that profile, list the available task ARNs, pick one and run execute-command:

$ aws sso login --profile my-sso-profile

Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request,
open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

XXXX-XXXX
Successfully logged into Start URL: https://my-sso.awsapps.com/start

$ aws ecs list-tasks \
  --desired-status RUNNING \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --profile my-sso-profile
{
    "taskArns": [
        "arn:aws:ecs:us-east-1: redacted:task/my-ecs-cluster/redacted",
        "arn:aws:ecs:us-east-1: redacted:task/my-ecs-cluster/redacted2"
    ]
}

$ aws ecs execute-command \
  --profile my-sso-profile \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --interactive \                                                                                  
  --task arn:aws:ecs:us-east-1:redacted:task/my-ecs-cluster/redacted \
  --command 'sh'

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0123456789


SessionId: ecs-execute-command-0123456789 : 
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error:
Failed to process action KMSEncryption: error while creating new KMS service, Error creating new aws sdk
session profile "my-sso-profile" is configured to use SSO but is missing required configuration:
sso_region, sso_start_url

Manually edit the ~/.aws/config file to add the reported missing configuration parameters:

@@ -7,6 +7,8 @@ region = us-east-1
 output = json
 [profile my-sso-profile]
 sso_session = my-sso-session
+sso_start_url = https://my-sso.awsapps.com/start
+sso_region = us-east-1
 sso_account_id = 123456789000
 sso_role_name = DevAccess
 region = us-east-1

Sign out then sign back into the profile and re-run the same ecs execute-command:

$ aws sso logout --profile my-sso-profile

$ aws sso login --profile my-sso-profile

Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request,
open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

XXXX-XXXX
Successfully logged into Start URL: https://my-sso.awsapps.com/start

$ aws ecs execute-command \
  --profile my-sso-profile \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --interactive \                                                                                  
  --task arn:aws:ecs:us-east-1:redacted:task/my-ecs-cluster/redacted \
  --command 'sh'

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-0123456789


SessionId: ecs-execute-command-0123456789 : 
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error:
Failed to process action KMSEncryption: Error calling KMS GenerateDataKey API: SSOProviderInvalidToken:
the SSO session has expired or is invalid caused by:
open /Users/me/.aws/sso/cache/a00fce5cb007c23a469c136160398db65edcb180.json: no such file or directory

Possible Solution

No response

Additional Information/Context

It appears the cache files generated by the SSO session profile and the legacy profile are using different values to generate the cache key.

$ rm -rf ~/.aws/sso/cache

$ aws sso login --profile my-sso-profile
# ...

$ ls -1 ~/.aws/sso/cache
b755b5ec73400c04400e978208d8559ad1f39053.json
# ...

$ aws sso logout --profile my-sso-profile

$ rm -rf ~/.aws/sso/cache

$ aws sso login --profile my-legacy-profile
# ...

$ ls -1 ~/.aws/sso/cache
a00fce5cb007c23a469c136160398db65edcb180.json
# ...

The SSO session profile generates the cache key from the session name while the legacy profile has the cache key generated from the URL:

Digest::SHA1.hexdigest("my-sso-session")
# => b755b5ec73400c04400e978208d8559ad1f39053

Digest::SHA1.hexdigest("https://my-sso.awsapps.com/start")
# => a00fce5cb007c23a469c136160398db65edcb180

The problematic ecs execute-command appears to ignore the sso_session configuration in the SSO profile and is looking for just sso_url; possibly why it complains about the missing configurations. When both sso_session and sso_url are present, the aws sso login gives higher precedence to the sso_session and uses that for the cache key which appears to be why after adding the "missing configuration" the ecs execute-command still fails, but now with a missing cache error.

CLI version used

aws-cli/2.11.20 Python/3.11.3 Darwin/22.4.0 source/arm64 prompt/off

Environment details (OS name and version, etc.)

macOS 13.3.1

[diegomosela]

I have the same problem.

[tim-finnigan]

Thanks for reaching out. @diegomosela which AWS CLI version are you using? Is this something that was working for you and @cupakromer before updating to a newer version of the AWS CLI? This may just be a limitation in the session manager plugin as reported here: aws/session-manager-plugin#58.

[diegomosela]

Thanks for the answer.

My AWS CLI version is 2.11.22 and the session manager plugin is 1.2.463.0.

I didn't get to use SSO and session-manager-plugin in a previous version, so I can't say at this point if it worked. I will try to downgrade the version to validate.

[cupakromer]

@tim-finnigan I do not see the same issue using the session manager directly. It connects just fine using the SSO profile.

$ session-manager-plugin --version
1.2.463.0


$ aws sso login --profile my-sso-profile

Attempting to automatically open the SSO authorization page...
...


$ aws ssm start-session --profile my-sso-profile --target ecs:my-ecs-cluster_redacted_redacted-123456


Starting session with SessionId: [email protected]
# 

[tim-finnigan]

@cupakromer I saw the original error you reported here in re:Post: https://repost.aws/questions/QUnv5hnNAXTeCRICHYRsDbJA/failing-in-kms-handshake. Can you validate your configuration per the documentation referenced there?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[cupakromer]

@tim-finnigan thank you for that link, unfortunately it's for a different error. The very end of the error messages are different and point to different issues.

My error:

Error creating new aws sdk session profile "my-sso-profile" is configured to use SSO but
is missing required configuration: sso_region, sso_start_url

Error from linked repost:

Error creating new aws sdk session LoadClientTLSCertError: client TLS cert(true) and key(false)
must both be provided

Further, as noted in the main description the error occurs when using SSO token provider configuration and the CLI process succeeds when using legacy non-refreshable configuration.

[rwky3gtelecoms]

Just encountered the same thing:

session-manager-plugin --version
1.2.463.0
aws --version
aws-cli/2.12.1 Python/3.11.3 Linux/5.10.0-23-amd64 docker/x86_64.ubuntu.20 prompt/off

Executing aws ssm start-session --target ecs:clustername_taskid_containerruntimeid does work though.

[ScubaDrew]

Having the issue on MacOS:

➜  ~ session-manager-plugin --version
1.2.463.0
➜  ~ aws --version
aws-cli/2.12.1 Python/3.11.4 Darwin/22.5.0 source/arm64 prompt/off
➜  ~ aws s3 ls --profile sso-gov-dev-01
2023-04-09 17:28:24 xxxx-tf-state-dev-01
2023-04-07 18:42:56 cloudtrail-xxxxx-d585-33cc-8502-xx
2023-05-25 20:02:07 dev-01-xxxx-1
➜  ~
➜  ~ aws ssm start-session --target i-xxxxxxxx --profile sso-gov-dev-01

Starting session with SessionId: [email protected]


SessionId: [email protected] :
----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: error while creating new KMS service, Error creating new aws sdk session profile "sso-gov-dev-01" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

➜  ~

[tim-finnigan]

It looks like everyone is using 1.2.463.0, maybe there was a recent change in that release causing this? If someone wants to try rolling back their version they could rule that out. I also reached out to the session manager team for more info and currently waiting to hear back.

[rwky3gtelecoms]

Rolled back to 1.2.398.0 from https://s3.amazonaws.com/session-manager-downloads/plugin/1.2.398.0/ubuntu_64bit/session-manager-plugin.deb works fine, so yep looks like the latest release broke it. Nice spot @tim-finnigan

[rwky3gtelecoms]

wait scratch that, it's not working (I forgot to switch to SSO). Same problem

----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: Error calling KMS GenerateDataKey API: SSOProviderInvalidToken: the SSO session has expired or is invalid

session-manager-plugin --version
1.2.398.0

[boskowski]

I'm experiencing the same issue across multiple environments: a clean Ubuntu 18.04, Windows 10, and WSL2. In each case, I encounter the following error when trying to start a session using AWS CLI:

----------ERROR-------
Encountered error while initiating handshake. KMSEncryption failed on client with status 2 error: Failed to process action KMSEncryption: Error calling KMS GenerateDataKey API: SSOProviderInvalidToken: the SSO session has expired or is invalid
caused by: open /home/user/.aws/sso/cache/058a9705fcdfbb9d98609a30f295c69b3034e687.json: no such file or directory

Here are the versions of session-manager-plugin and AWS CLI I'm using:

$ session-manager-plugin --version
1.2.463.0

$ aws --version
aws-cli/2.13.5 Python/3.11.4 Linux/5.4.0-150-generic exe/x86_64.ubuntu.18 prompt/off

also tried with

aws-cli/2.13.5 Python/3.11.4 Linux/5.15.90.1-microsoft-standard-WSL2 exe/x86_64.debian.11 prompt/off
aws-cli/1.29.15 Python/3.8.10 Windows/10 botocore/1.31.15

I've attempted to resolve the issue by refreshing the SSO login and verifying the SSO configuration, but the problem persists. If anyone has encountered this issue and found a workaround, your insights would be greatly appreciated.

[cupakromer]

@boskowski The workaround I am using is to configure a profile using only the legacy non-refreshable configuration and use that for the command.

I have two separate profiles configured, one using that legacy non-refreshable configuration structure which I use just for this command and another using the newer SSO token provider configuration which I use for pretty much everything else.

[tim-finnigan]

It looks like another session-manager-plugin version was released (1.2.497.0): https://docs.aws.amazon.com/systems-manager/latest/userguide/plugin-version-history.html.

If anyone is still encountering issues reported here after upgrading then I recommend creating an issue in this repo for further tracking: https://github.com/aws/session-manager-plugin/issues

[rwky3gtelecoms]

It works for me with the latest version

session-manager-plugin --version
1.2.497.0

[tim-finnigan]

Thanks @rwky3gtelecoms for letting us know! I'll go ahead and close this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.