AWS assume role not working

[hashibot]

This issue was originally opened by @hkalyana as hashicorp/terraform#11270. It was migrated here as part of the provider split. The original body of the issue is below.

---

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version -0.8.0

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.

Affected Resource(s)

Please list the resources as a list, for example:

• aws_provider

Copy-paste your Terraform configurations here - for large Terraform configs,

provider "aws"
{
    region = "${var.region}"
    assume_role {
    role_arn = "arn:aws:iam::<trustedaccount>:role/trustedrole"
    }
 }

Expected Behavior

Expected the resources to be created on the trusting account

Actual Behavior

Received error The role "arn:aws:iam:::role/trustedrole" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Created iam role in the trusting account with the necessary permission .
2. Created a policy on the trusted account with sts assume role permission and pointing the arm of the trusting account in the resource
   3)Attached the policy to the user
   4)Used the same role ARN of the trusting account in the terraform assume role

Important Factoids

The procedure I followed is working for me when I try to switch the roles in AWS GUI and I am able to create resources on the trusting account .

References

[catsby]

Hello – I'm attempting to reproduce this and confirm if it's a Terraform issue or policy issue. Right now I have a configuration that demonstrates Terraform is successfully able to assume a role and create a resource, so I was hoping someone could examine my configuration and see if I've misconfigured it and not correctly reproducing your situation.

Below is my config. I have IAM admin credentials exported in my environment, but they are not AWS Root credentials. Role terraform_11270 (named from the original terraform issue) is a role that has permissions to work on s3 and ec2 (if I understand IAM policies correctly, please correct me if I’m wrong).

# Grab the ARN of the current logged in user
data "aws_caller_identity" "current" {}

# create a role which allows the current user to assume it
resource "aws_iam_role" "terraform_11270" {
  name = "terraform_11270"
  path = "/test/"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "${data.aws_caller_identity.current.arn}"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "terraform_11270" {
  name = "terraform_11270"
  role = "${aws_iam_role.terraform_11270.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*",
        "ec2:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

# configure this provider alias to only use the IAM Role created above
provider "aws" {
  alias = "iamrole"

  assume_role {
    role_arn = "${aws_iam_role.terraform_11270.arn}"
  }
}

resource "aws_security_group" "primary" {
  name = "primary"
}

# Create a security group with the above IAM Role assumed
resource "aws_security_group" "secondary" {
  provider = "aws.iamrole"
  name     = "secondary"
}

With this configuration I was able to create the security group secondary by assuming the IAM role. Can anyone examine this configuration and let me know what I’m missing?

[catsby]

I'm going to go ahead and close this issue for now. With the config above I've demonstrated that assume_role works at the provider level, assuming the correct policies are in place.

If you feel I've demonstrated the working behavior incorrectly, please let me know and we'll reopen for another look!

[ashb]

It works for non-MFA roles, yes. For roles enforcing MFA-protected access something like hashicorp/terraform#11734 (which hasn't been migrated?) is needed.

[FransUrbo]

This doesn't work at all for me.

Files

~/.aws/credentials

[root]
# Account: <account_id0>
aws_access_key_id = SOMETHING
aws_secret_access_key = SECRET

[core]
aws_access_key_id = SOMETHING_ELSE
aws_secret_access_key = OTHER_SECRET

#[test]
# Account: <account_id1>
#aws_access_key_id = YET_ANOTHER
#aws_secret_access_key = MORE_SECRETS

The test profile here disabled, so that TF don't accidentally pick it up somehow. But with that enabled, and a different entry in the profile.tf file for that, it works. But that's not doing assume_role..

remote.tf

data "terraform_remote_state" "base" {
  backend   = "s3"
  config {
    bucket  = "my-root-account-bucket"
    key     = "terraform-base.tfstate"
    region  = "eu-west-1"
    profile = "root"
  }
}

data "terraform_remote_state" "core" {
  backend   = "s3"
  config {
    bucket  = "my-root-account-bucket"
    key     = "terraform-core.tfstate"
    region  = "eu-west-1"
    profile = "root"
  }
}

data "terraform_remote_state" "test" {
  backend   = "s3"
  config {
    bucket  = "my-root-account-bucket"
    key     = "terraform-test.tfstate"
    region  = "eu-west-1"
    profile = "root"
  }
}

provider.tf

provider "aws" {
  region  = "eu-west-1"
  profile = "core"
  alias   = "core"
}

provider "aws" {
  region  = "eu-west-1"
#  profile = "test"
  assume_role {
    role_arn = "arn:aws:iam::<account_id1>:role/Cross_Account"
  }
}

The second provider will then be the "default" one and the first being referenced in certain places as provider = "aws.core"..

This gives me:

[Turbo-Fredrikssons-MacBook-Pro]$ terraform plan -no-color
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.test: Refreshing state...
data.terraform_remote_state.core: Refreshing state...
data.terraform_remote_state.base: Refreshing state...
data.aws_caller_identity.core: Refreshing state...
Error refreshing state: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

I had wanted this to work:

~/.aws/credentials

[root]
# Account: <account_id0>
aws_access_key_id = SOMETHING
aws_secret_access_key = SECRET

[test]
role_arn = arn:aws:iam::<account_id1>:role/Cross_Account
source_profile = core
mfa_serial = arn:aws:iam::<account_id0>:mfa/turbo

but that don't work either... Not even if I disable MFA and remove the mfa_serialline.

[FransUrbo]

Some small change. If I duplicate the [core] profile into a [defaults] profile, I instead get:

[Turbo-Fredrikssons-MacBook-Pro]$ terraform plan -no-color
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.core: Refreshing state...
data.terraform_remote_state.test: Refreshing state...
data.terraform_remote_state.base: Refreshing state...
data.aws_caller_identity.core: Refreshing state...
Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::<account_id1>:role/Cross_Account" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Neither of these reasons seems valid, because the ARN to the role was copy-and-pasted from the IAM console, I can successfully go from my root account (as in, not root credentials, but the account I've labeled "root") to this test account. And I disabled MFA requirements on the role..

[FransUrbo]

@catsby I think the problem with your example is

1. You have your "base" credentials, which you use to create a role and the primary SG, both in the "base" account
2. You're creating a role in your "base" account, which you then use to create the secondary SG (also in the "base" account ??).

But where is that SG created? In another account? Or the same one? Could the problem I'm having is that I have problem with some cross-account-assume-role thingiemajig??

[Ralf-Te]

Hello @catsby,
As @FransUrbo describes an issue is still persisting for him. I am looking for the same scenario as he does, meaning assuming a role of a second account.
For me the question arises if the feature is intended to be used for cross account access or not.
I think a lot of people expect this, but from your test scenario it is not intended, because the test does not cover it.
Can you please clear this up?

[adventureisyou]

Same issue here. Our IAM users are managed through a central account, requiring MFA. Users have barely any permissions in the central account, but can assume roles in other accounts managed through AWS Organisations. I think this is a pretty common setup, especially at larger companies, and is the recommended best practice by AWS themselves.

Even if I avoid using the aws credentials and config files by embedding my credentials straight into the tf file like this:

provider "aws" {
  region = "eu-west-1"
  access_key = "REDACTED" # Central account
  secret_key = "REDACTED" # Central account
  token = "arn:REDACTED" # Central account
  assume_role {
    role_arn = "arn:REDACTED" # Satellite account
  }
}

I get the following error:

2017/07/05 10:22:19 [DEBUG] plugin: terraform: aws-provider (internal) 2017/07/05 10:22:19 [DEBUG] plugin: waiting for all plugin processes to complete...
* provider.aws: The role "arn:REDACTED" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

My guess is that this is because the MFA code isn't being requested and sent before trying to assume the role (the trust policy on the role requires that the user is MFA authenticated).

This is a real blocker for us. Let me know if there's anything else I can provide to help.

[FransUrbo]

@DrSolaris Could you, temporarily, disable MFA on the satellite account role and see if that makes any difference?

The reason for this, is that I suspect that there's TWO problems here - TF can't assume remote roles AND it have problems with MFA. But I'm not sure...

[amvapor]

I'm running into this case now as well, I have multiple satellite accounts, ones with MFA fail to work with TF whereas ones without MFA security are able to be provisioned with TF. This is currently requiring the use of kludgy wrapper scripts to make work.

[et304383]

The terraform files in my opinion should not be specifying anything to do with authentication. Role names are different and if it's an admin with an admin role or a developer with a developer role running a terraform file they should not have to edit it to specify the profile/role/etc to use. This should be input to terraform.

Currently, the only way I can, as an admin with MFA on my role/profile, give credentials properly to a terraform file is to have an external script issue sts assume role calls and then set environment variables to the temporary access key and secret. This is not an efficient way to pass in credentials. The GO SDK fully supports MFA + assume role style profiles (with role_arn and source_profile) so terraform should delegate authentication calls to the GO SDK.

I hoped there would be a --profile option to terraform but barring that setting the env var AWS_PROFILE to my profile name should be the only thing I need to do on the cli prior to calling terraform.

In my opinion.

[FransUrbo]

@catsby Could you please reopen this again? It's clearly not fixed (satisfactory)...

[artkrz]

This is clearly not fixed as other mentioned. Does not work at all with MFA!

[JoshCooley]

@FransUrbo, one quick correction of the ~/.aws/credentials file you posted under "I had wanted this to work". Those [test] profile parameters should be in the ~/.aws/config file instead, and the profile name should be prefixed with "profile" (this prefix is only needed in the ~/.aws/config file), e.g. :

~/.aws/config:

[profile test]
role_arn = arn:aws:iam::<account_id1>:role/Cross_Account
source_profile = core
mfa_serial = arn:aws:iam::<account_id0>:mfa/turbo

This is how my config/credentials files are configured, and role switching is working correctly in the AWS CLI and Terraform, but then again I'm not using MFA.

[FransUrbo]

@JoshCooley What's the difference? The aws command can handle the credentials file just fine...

[et304383]

So even profiles without MFA that are using source profile don't work with AWS_PROFILE=<profile name>

The Go SDK fully supports source profile/role arn profiles and MFA. Hashicorp, what needs to be done to support this in terraform?

[choppedpork]

@et304383 @FransUrbo #1608 might be of interest. If my understanding of the latest comments is correct, this will require a release of the provider for aws resources but also a release of terraform for remote state kept in aws s3.

[et304383]

@choppedpork so how do we get this to happen?

[farmerbean]

Happy New Year! Anyone.. anyone.. Bueller?

[rdonkin]

I spent some time trying to get Terraform to assume roles in a cross-account setup, trying various workarounds.

The real solution in the end was to use Terragrunt, which worked first time using this feature - Terragrunt does the assume-role, bypassing the Terraform issues in this area - just give Terragrunt a role ARN.

[catsby]

Hello –

This issue has gone several ways since it opened, I think. I was able to create an EC2 instance with a profile (named cts_base_instance_role), ssh into it, install Terraform, and then use it's instance profile to assume another role (named cts_subuser_role) with just this provider block in the configuration file on the ec2 instance:

provider "aws" {
  region = "us-west-2"
  assume_role {
      role_arn = "arn:aws:iam::XXXXXXX:role/cts_subuser_role"
  }
}

Terraform correctly used the IAM profile creds to assume the role. The key bits:

• the instance profile named cts_base_instance_role had this policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {"AWS": "*"},
            "Effect": "Allow",
            "Sid": ""
        }
    ]
}

• cts_subuser_role does not have the "require MFA" stipulation. I did not have MFA on the IAM instance profile, and having that condition made things predictably fail
• cts_subuser_role needed to explicitly allow the instance profile to assume it with this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::XXXXXX:role/cts_base_instance_role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This does not demonstrate cross account role. I does demonstrate Terraform picking up EC2 role creds and then using those creds to assume the role. I realize this isn't explicitly what others have reported, but I believe it was one instance and it demonstrates assume_role working.

If you have another auth issue that is not covered by what I describe above, please open a new issue and detail what we're missing or where a bug happens and we'll take a fresh look.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!