AssumeRole profile not prompting for MFA when source profile uses credential_process

[MrGossett]

$ aws --version
aws-cli/1.14.9 Python/3.6.3 Darwin/17.3.0 botocore/1.8.13

I have a profile using the credential_process config value to fetch credentials from the macOS keychain. The config for this profile is:

# ~/.aws/config
[profile user]
region = us-east-1
credential_process = aws keychain user

The keychain alias is defined as follows:

# ~/.aws/cli/alias

keychain =
  !f() {
    ACCESS_KEY=$(security find-generic-password -s "aws $1 access" -w login.keychain)
    SECRET_KEY=$(security find-generic-password -s "aws $1 secret" -w login.keychain)
    echo \{\"AccessKeyId\":\""$ACCESS_KEY"\",\"SecretAccessKey\":\""$SECRET_KEY"\",\"Version\":1}
  }; f

I took this directly from the awesome presentation from re:Invent 2017 by @kyleknap . Thanks Kyle :)

This part of my config is working perfectly.

$ aws configure list --profile user
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                     user           manual    --profile
access_key     ****************XXXX   custom-process
secret_key     ****************XXXX   custom-process
    region                us-east-1              env    AWS_DEFAULT_REGION

$ aws sts get-caller-identity --profile user
{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/mrgossett"
}

I have an AssumeRole profile configured with a role_arn and a mfa_serial, and using the first profile as its source:

[profile role]
region = us-east-1
role_arn = arn:aws:iam::210987654321:role/SomeRole
mfa_serial = arn:aws:iam::123456789012:mfa/mrgossett
source_profile = user

Using this AssumeRole profile, the CLI is not prompting me for an MFA code, and appears to fail before calling STS AssumeRole.

$ aws sts get-caller-identity --profile role --debug
2017-12-12 22:14:38,341 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.14.9 Python/3.6.3 Darwin/17.3.0 botocore/1.8.13
2017-12-12 22:14:38,342 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'get-caller-identity', '--profile', 'role', '--debug']
2017-12-12 22:14:38,342 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x1080cd268>
2017-12-12 22:14:38,342 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,342 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x107bc1d90>
2017-12-12 22:14:38,342 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,343 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2017-12-12 22:14:38,345 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,345 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2017-12-12 22:14:38,345 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x107e71840>
2017-12-12 22:14:38,345 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,345 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,345 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-12-12 22:14:38,346 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/data/sts/2011-06-15/service-2.json
2017-12-12 22:14:38,347 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.sts: calling handler <function register_retries_for_service at 0x1077649d8>
2017-12-12 22:14:38,347 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: sts
2017-12-12 22:14:38,348 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <function add_waiters at 0x1080d5488>
2017-12-12 22:14:38,356 - MainThread - awscli.clidriver - DEBUG - OrderedDict()
2017-12-12 22:14:38,356 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_streaming_output_arg at 0x1080cd510>
2017-12-12 22:14:38,356 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_cli_input_json at 0x107bcc598>
2017-12-12 22:14:38,356 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function unify_paging_params at 0x108044ea0>
2017-12-12 22:14:38,364 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/data/sts/2011-06-15/paginators-1.json
2017-12-12 22:14:38,364 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_generate_skeleton at 0x1080317b8>
2017-12-12 22:14:38,364 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1081ca908>>
2017-12-12 22:14:38,364 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1081ca940>>
2017-12-12 22:14:38,365 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.cli-input-json: calling handler <function uri_param at 0x107b539d8>
2017-12-12 22:14:38,365 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.generate-cli-skeleton: calling handler <function uri_param at 0x107b539d8>
2017-12-12 22:14:38,365 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1081ca908>>
2017-12-12 22:14:38,365 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1081ca940>>
2017-12-12 22:14:38,365 - MainThread - botocore.session - DEBUG - Loading variable region from environment with value 'us-east-1'.
2017-12-12 22:14:38,366 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,366 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2017-12-12 22:14:38,366 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value 'role'.
2017-12-12 22:14:38,366 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-12-12 22:14:38,366 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-12-12 22:14:38,366 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 347, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 519, in __call__
    call_parameters, parsed_globals)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 637, in invoke
    verify=parsed_globals.verify_ssl)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/session.py", line 850, in create_client
    credentials = self.get_credentials()
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/session.py", line 474, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1594, in load_credentials
    creds = provider.load()
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1163, in load
    return self._load_creds_via_assume_role(self._profile_name)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1172, in _load_creds_via_assume_role
    role_config, profile_name
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1313, in _resolve_source_credentials
    return self._resolve_credentials_from_profile(source_profile)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1322, in _resolve_credentials_from_profile
    return self._load_creds_via_assume_role(profile_name)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1170, in _load_creds_via_assume_role
    role_config = self._get_role_config(profile_name)
  File "/usr/local/Cellar/awscli/1.14.9/libexec/lib/python3.6/site-packages/botocore/credentials.py", line 1215, in _get_role_config
    role_arn = profile['role_arn']
KeyError: 'role_arn'
2017-12-12 22:14:38,369 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

'role_arn'

However, if I remove the keychain alias and instead write the credentials to ~/.aws/credentials, it starts working again.

$ aws configure --profile user
AWS Access Key ID [None]: [snip]
AWS Secret Access Key [None]: [snip]
Default region name [us-east-1]:
Default output format [None]:

$ aws configure list --profile user
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                     user           manual    --profile
access_key     ****************XXXX shared-credentials-file
secret_key     ****************XXXX shared-credentials-file
    region                us-east-1              env    AWS_DEFAULT_REGION

$ cat ~/.aws/config | head -n 3
[profile user]
region = us-east-1
# credential_process = aws keychain mlt

$ aws sts get-caller-identity --profile role
Enter MFA code for arn:aws:iam::123456789012:mfa/mrgossett:
{
    "UserId": "AROAXXXXXXXXXXXXXXXXX:botocore-session-1513135483",
    "Account": "210987654321",
    "Arn": "arn:aws:sts::210987654321:assumed-role/SomeRole/botocore-session-1513135483"
}

I've looked through issues here in GitHub following the release of v1.12.0 on November 22 when the credential_process feature was introduced, and I can't find anything similar. I think I may have hit some edge case with the combo of credential_process on a source profile and mfa_serial on an AssumeRole profile...?

[joguSD]

Unfortunately this isn't possible at the moment.
Currently, the source_profile needs to point to a profile that is either another assume role or static credentials. There's an upstream feature request for this in botocore: boto/botocore#1329

Closing in favor of that issue.