amazon: Packer cannot assume role with credential_source

[Asura9]

I use AWS CodeBuild to launch AMI building process with Packer, in a cross-account context.
In a central account, Codebuild project is started with a specific role, then we need to make a chained assume role (short term) and let packer manage this assume-role.
But in this way, build duration is limited to ~1 hour as short term credentials can't last more than 1 hour.

Before moving to a non-cross account architecture, we want to test every possibilities provided by Packer.
One of those is using the newly integrated feature in AWS Go SDK, credential_source, but it does not seems to work with Packer.

We execute a shell script before running packer, to configure AWS profiles.
This is how the working assume role is done, but we can't renew credentials, and bypass 1 hour limitation, even if container credentials have been renew by CodeBuild in metadata (~/.aws/config and ~/.aws/credentials files are created successfully after that):

  AppRoleArn="arn:aws:iam::XXXXXXXXXXX:role/rol-test-dev"
  curl -sqL -o aws_credentials.json http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI > aws_credentials.json
  aws configure set region "eu-west-1"
  aws configure set profile.codebuild.aws_access_key_id `jq -r '.AccessKeyId' aws_credentials.json`
  aws configure set profile.codebuild.aws_secret_access_key `jq -r '.SecretAccessKey' aws_credentials.json`
  aws configure set profile.codebuild.aws_session_token `jq -r '.Token' aws_credentials.json`
  aws configure set profile.codebuild.expiration `jq -r '.Expiration' aws_credentials.json`
  aws configure set profile.packer.role_arn "${AppRoleArn}"
  aws configure set profile.packer.source_profile "codebuild"
  export AWS_PROFILE=packer

Here are the packer Codebuild logs when its working :

·[1;32mamazon-ebs output will be in this color.·[0m
 ·[1;32m==> amazon-ebs: Prevalidating AMI Name: XXXXXXXXXXXXXXXX·[0m
·[0;32m amazon-ebs: Found Image ID: ami-XXXXXXXXXXXXXXX·[0m
·[1;32m==> amazon-ebs: Creating temporary keypair: packer_5bd96a1d-1701-adc8-9575-70006aa0e5d2·[0m
·[1;32m==> amazon-ebs: Launching a source AWS instance...·[0m
·[1;32m==> amazon-ebs: Adding tags to source instance·[0m
·[0;32m amazon-ebs: Instance ID: i-XXXXXXXXXXXXXX·[0m
·[1;32m==> amazon-ebs: Waiting for instance (i-XXXXXXXXXXXXX) to become ready...·[0m
·[1;32m==> amazon-ebs: Using ssh communicator to connect: XXXXXXXXXXXXXX·[0m
·[1;32m==> amazon-ebs: Waiting for SSH to become available...·[0m

This is the not working assume role, the ~/.aws/config file is generated successfully:

  AppRoleArn="arn:aws:iam::XXXXXXXXXXX:role/rol-test-dev"
  aws configure set region "eu-west-1"
  aws configure set profile.packer.role_arn "${AppRoleArn}"
  aws configure set profile.packer.credential_source "EcsContainer"
  export AWS_PROFILE=packer

Here are the packer CodeBuild logs when its not working :

·[1;32mamazon-ebs output will be in this color.·[0m
 ·[1;32m==> amazon-ebs: Prevalidating AMI Name: XXXXXXXXXXXXX·[0m
·[0;32m amazon-ebs: Found Image ID: ami-XXXXXXXXXXXX·[0m
·[1;31m==> amazon-ebs: Describing the subnet: subnet-XXXXXXXXXXXXX returned error: InvalidSubnetID.NotFound: The subnet ID 'subnet-XXXXXXXXXXXXXX' does not exist
==> amazon-ebs: status code: 400, request id: 78cb2098-32d5-4cb6-a34f-1fc84f32257b.·[0m
·[1;31mBuild 'amazon-ebs' errored: Describing the subnet: subnet-XXXXXXXXXXXXXXX returned error: InvalidSubnetID.NotFound: The subnet ID 'subnet-XXXXXXXXXXXXX' does not exist
 status code: 400, request id: 78cb2098-32d5-4cb6-a34f-1fc84f32257b.·[0m
 ==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Describing the subnet: subnet-XXXXXXXXXXXXXXX returned error: InvalidSubnetID.NotFound: The subnet ID 'subnet-XXXXXXXXXXXXXXX' does not exist
 status code: 400, request id: 78cb2098-32d5-4cb6-a34f-1fc84f32257b.
 ==> Builds finished but no artifacts were created.

I tried to remove that line and rebuild on @rickard-von-essen advice, but I still get the same error
https://github.com/hashicorp/packer/blob/3315812c2c96df0d5bee903073b87b8798d51246/builder/amazon/common/access_config.go#L75

Packer version tested 1.3.2
CodeBuild ECS Container aws/codebuild/eb-python-2.7-amazonlinux-64:2.1.6

[karl-cardenas-coding]

v1.3.4

As everyone else on this thread, I too encounter this issue in a pipeline, however when I invoke Packer from my workstation the assumed role and respective target account is found. Just a regular config and credentials file, nothing out of the ordinary.

[SwampDragons]

@karl-cardenas-coding Are you also using CodeBuild?

[karl-cardenas-coding]

@SwampDragons No, I'm using Gitlab and Gitlab CI

[mfunger]

v 1.3.4
Building an AMI using Jenkins and an ECS worker. The Jenkins worker assumes a role in a separate account that has all the requisite IAM policies, but I still get:

Debug mode enabled. Builds will not be parallelized.
[1;32mamazon-ebs output will be in this color.[0m

[1;31mBuild 'amazon-ebs' errored: error validating regions: UnauthorizedOperation: You are not authorized to perform this operation.
	status code: 403, request id: cbd552b9-d9e2-49c2-a23a-fc76ae6dd655 [0m

[rickard-von-essen]

@mfunger that seems unrelated and like a configuration error. Send a email to the mailing list with some more details.

[cloudlena]

I have just run into a very similar bug. It seems, using profiles only works if the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are not set. I just did a test where I had the config files set up and the two mentioned environment variables set. In that case, the aws CLI worked as expected but packer didn't. When I unset the two environment variables, all of a sudden, packer started to work. That was with packer version 1.3.5. Is there maybe a bug in packer handling these variables and the config files in combination?

[rickard-von-essen]

@mastertinner that is something unrelated, and seems to be correct according to the documentation. If it still seems wrong after reading that please open a new issue.

[sc250024]

All,

I'm almost positive this is an underlying problem with the AWS SDK for Go. See these issues / pull requests:

• aws/session: Add support for using role_arn on ec2 with instance profile aws/aws-sdk-go#1019
• aws/session: Adding support for credential_source aws/aws-sdk-go#2201 => This was the PR that should have fixed this
• Support credentialsSource config and environment option aws/aws-sdk-go-v2#274

I am also using GitLab CI, and I'm almost positive because using the aws CLI, I do everything as specified. I have three accounts: tooling, preprod, and prod. The GitLab runners are in Kubernetes in the tooling account, and this serves as the default credential source in the AWS config file. The pod has permissions to assume a role using Kube2IAM.

Packer version

$ packer version
Packer v1.4.2

I have the following AWS config in my CI image:

~/.aws/config

[default]
credential_source=Ec2InstanceMetadata
output=json
region=eu-west-1
role_arn=arn:aws:iam::<tooling-account-id>:role/ToolingGitLabRunner

[profile preprod]
output=json
region=eu-west-1
role_arn=arn:aws:iam::<preprod-account-id>:role/PreprodGitLabRunner
source_profile=default

[profile prod]
output=json
region=eu-west-1
role_arn=arn:aws:iam::<prod-account-id>:role/ProdGitLabRunner
source_profile=default

I can grab credentials for base IAM role ToolingGitLabRunner using cURL:

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ToolingGitLabRunner | jq

{
  "AccessKeyId": "<some-key>",
  "Code": "Success",
  "Expiration": "<some-timestamp>",
  "LastUpdated": "<some-other-timestamp>",
  "SecretAccessKey": "<some-secret-access-key>",
  "Token": "<some-token>",
  "Type": "AWS-HMAC"
}

I can list S3 buckets from the command line inside of the job pod:

$ aws --profile prod s3 ls
...
2019-07-12 19:55:06 some-test-bucket-4wxow5tv
...

For a job pod (say, building packer templates for the prod account), here's what's configured:

$ env | grep -i aws
AWS_PROFILE=prod
AWS_SDK_LOAD_CONFIG=1

But running the Packer build doesn't work:

$ packer build aws/my-template.json
amazon-ebs output will be in this color.

Build 'amazon-ebs' errored: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::<prod-account-id>:role/ProdGitLabRunner, source profile has no shared credentials

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::<prod-account-id>:role/ProdGitLabRunner, source profile has no shared credentials

==> Builds finished but no artifacts were created.

[Asura9]

Hello,

I tested today with packer 1.4.3, and it works properly now.

$ aws configure set profile.packer.role_arn "${AppRoleArn}"
$ aws configure set profile.packer.credential_source "EcsContainer"
$ export AWS_PROFILE=packer

Then packer build command is able to assume the role named packer, and is also able to regenerate short-term credentials.

[SwampDragons]

Great to hear! Marking this issue as closed. If anyone has further issues, reach out and we can open a new issue with new details.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.