iam_role inline_policy: Issue importing iam role with inline policies wanting to be deleted and recreated when there is no difference

[klear-nasuni]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.5
AWS provider version 3.35.0

Affected Resource(s)

• iam_role inline_policy

Terraform Configuration Files

data "aws_iam_policy_document" "ec2_assume_role_policy_document" {
  statement {
    effect = "Allow"

    principals {
      type = "Service"
      identifiers = [
        "ec2.amazonaws.com",
      ]
    }
    actions = [
      "sts:AssumeRole",
    ]
  }
}

resource "aws_iam_role" "ec2_role" {
  name               = "Test-terraform-import-role"
  assume_role_policy = data.aws_iam_policy_document.ec2_assume_role_policy_document.json
  inline_policy {
    name   = "Testinlinepolicy"
    policy = data.aws_iam_policy_document.ec2_role_policy_document.json
  }
}

data "aws_iam_policy_document" "ec2_role_policy_document" {
  statement {
    sid    = "VisualEditor0"
    effect = "Allow"
    actions = [
      "ec2:DescribeScheduledInstances",
      "ec2:DescribeScheduledInstanceAvailability",
      "ec2:DescribeFastSnapshotRestores",
      "ec2:DescribeElasticGpus"
    ]
    resources = [
      "*",
    ]
  }
}

Expected Behavior

When running import the existing role described in terraform above via the command below, it should import the role without requiring deleting/recreating policies.

terraform import aws_iam_role.ec2_role Test-terraform-import-role

Actual Behavior

The plan lists that the inline_policy must be destroyed and recreated though they are the same (see plan below). This is an issue for potential downtime of a system when the inline policy is being deleted and recreated.

  ~ resource "aws_iam_role" "ec2_role" { 
        ... other attributes of resource that don't require changes ....
        
      - inline_policy {
          - name   = "Testinlinepolicy" -> null
          - policy = jsonencode(
                {
                  - Statement = [
                      - {
                          - Action   = [
                              - "ec2:DescribeScheduledInstances",
                              - "ec2:DescribeScheduledInstanceAvailability",
                              - "ec2:DescribeFastSnapshotRestores",
                              - "ec2:DescribeElasticGpus",
                            ]
                          - Effect   = "Allow"
                          - Resource = "*"
                          - Sid      = "VisualEditor0"
                        },
                    ]
                  - Version   = "2012-10-17"
                }
            ) -> null
        }
      + inline_policy {
          + name   = "Testinlinepolicy"
          + policy = jsonencode(
                {
                  + Statement = [
                      + {
                          + Action   = [
                              + "ec2:DescribeScheduledInstances",
                              + "ec2:DescribeScheduledInstanceAvailability",
                              + "ec2:DescribeFastSnapshotRestores",
                              + "ec2:DescribeElasticGpus",
                            ]
                          + Effect   = "Allow"
                          + Resource = "*"
                          + Sid      = "VisualEditor0"
                        },
                    ]
                  + Version   = "2012-10-17"
                }
            )
        }

  }

Steps to Reproduce

1. Manually create resource that is described in the terraform code above
2. terraform init
3. terraform import aws_iam_role.ec2_role Test-terraform-import-role
4. terraform plan -out=plan.out to see the error

Important Factoids

This is also an issue with inline policies created outside of terraform without an Sid as the terraform provider requires that a policy has this (it will set it to be sid="" if there is not one). This might be another issue that is causing the problem.

Appears that issue is with code here in the provider coming from this library.

[stevehipwell]

This appears to be related to #19349 & #19245.

And the linked library now has an opened issue jen20/awspolicyequivalence#10.

[teddylear]

@stevehipwell I'm not sure this is an issue with that library as I originally thought. I was able to add a test changing the order of the actions listed in the policy and the library showed the policies were equivalent. Testing this again with the latest version of the aws provider with a debug message just before the equivalency call, I saw that two calls were made to the library both with a policy being compared to a nil object. This leads me to believe the issue is that the in this case something is wrong with the state not comparing the the policies and assuming a delete then recreate needs to occur.

[stevehipwell]

@teddylear my original thoughts before seeing this was that the state comparison was comparing computed values as this often shows up with the content of jsonencode showing as being removed with the replacement known after apply.

[celik0311]

Any recommended workaround for this issue? Importing many roles with inline_policies and spot-checking them to make sure they are indeed the same is tedious.

[teddylear]

@celik0311 I haven't found a workaround for this yet. Even if they inline_policy is the same as the one in AWS, it's still producing a diff. I don't know if this is an issue with terraform itself or how inline policies on an iam role are being compared in state.

[celik0311]

Bummer, thanks for the reply @teddylear

[tsub]

I found a workaround for this problem.

After importing the aws_iam_role resource, I checked the contents of the terraform.tfstate file and found it to be in the following format.

"inline_policy": [
    {
        "name": "Testinlinepolicy",
        "policy": "{\"Statement\":[{\"Action\":[\"ec2:DescribeScheduledInstances\",\"ec2:DescribeScheduledInstanceAvailability\",\"ec2:DescribeFastSnapshotRestores\",\"ec2:DescribeElasticGpus\"],\"Effect\":\"Allow\",\"Resource\":\"*\",\"Sid\":\"VisualEditor0\"}]}"
    }
],

The above value is then used as is for the policy value to resolve the difference in terraform plan.

  inline_policy {
    name   = "Testinlinepolicy"
    policy = "{\"Statement\":[{\"Action\":[\"ec2:DescribeScheduledInstances\",\"ec2:DescribeScheduledInstanceAvailability\",\"ec2:DescribeFastSnapshotRestores\",\"ec2:DescribeElasticGpus\"],\"Effect\":\"Allow\",\"Resource\":\"*\",\"Sid\":\"VisualEditor0\"}]}"
  }

Apparently, the policy value after terraform import does not include the newline character (\n), so the difference is probably due to the presence of the newline character.

However, the downside is that it makes Terraform's code very hard to read 😅.

[celik0311]

to add more context, I am using a JSON file and defining the policy statement as

policy = file("foo.json")

The policy is an exact JSON representation of what is in AWS, and after importing the IAM role with that inline policy, it still sees a diff. This is also the case for an inline policy associated with an iam role that has been imported using terraform.

My thinking is it might be worth looking at the pkg that's being used to determine that.

terraform-provider-aws/aws/diff_suppress_funcs.go

Line 16 in ac06ced

equivalent, err := awspolicy.PoliciesAreEquivalent(old, new)

which uses the below pkg

https://github.com/jen20/awspolicyequivalence

[celik0311]

so after some tinkering, I've found what I think is causing this. If you look at the state file after doing the import you will see the policy defined as a string include "hidden" chars (newline, spaces..). I was able to match my local policy with the one defined in state and got the desired result. I suspect the policy comparison lib includes spaces, newlines maybe tabs in the as diffs. This means that if your policy that's been imported does not end with a newline but the one you have defined in terraform does it will be considered different, and thus terraform will try to recreate it. My expected behavior would be that if 2 policies are the same except one ends in a newline and the other does not they are still the same. Hope this helps.

[celik0311]

Well even after importing successfully, any change to an inline policy triggers a recreation of that inline policy :/

[github-actions]

This functionality has been released in v3.69.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[danopia]

Well even after importing successfully, any change to an inline policy triggers a recreation of that inline policy :/

I've noticed this too, did you find a solution? It seems like inline policies are basically unusable for anything that will change because the terraform plan outputs a recreation diff on every change. The IAM API supposedly supports upserting so Terraform shouldn't need to remove the existing inline policy.

I found a workaround of using the aws_iam_role_policy resource, but this doesn't work in cases where Terraform should be exclusively managing inline policies (as in, removing out-of-band policies)

[klear-nasuni]

@danopia Does this still happen even with the latest version that was suppose to fix this functionality? If so I'll reopen ticket (or contact someone to do so).

[danopia]

It looks like your original issue was:

The plan lists that the inline_policy must be destroyed and recreated though they are the same

This looks fixed. But then you someone else described how the recreation still happens if the inline policy is different, and I see that too. So I suppose that's worthy of a new ticket. I'll go file one.

I'm running the latest provider (3.70)

[danopia]

Sorry, the comment I originally replied to was from @celik0311, I apologize for the false "you"

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.