AWS EMR: Changes to EC2 instance attributes are not applied properly

[ddcprg]

Terraform Version

Terraform 0.11.7 and earlier
AWS provider 1.14.1 and earlier

Affected Resource(s)

• aws_emr_cluster

Terraform Configuration Files

Initial config:

resource "aws_emr_cluster" "ad-hoc-query-cluster" {
  name                 = "ad-hoc-query-cluster"
  release_label        = "emr-5.12.0"
  applications         = ["Hadoop", "Hive", "HCatalog", "Spark"]
  log_uri              = "s3n://my-bucket/emr/logs/ad-hoc-query-cluster/"
  master_instance_type = "m3.xlarge"
  core_instance_type   = "m3.xlarge"
  core_instance_count  = 2
  service_role         = "EMR_DefaultRole"

  ec2_attributes {
    key_name                          = "my-pem-file"
    subnet_id                         = "subnet-123456"
    emr_managed_master_security_group = "sg-123456"
    emr_managed_slave_security_group  = "sg-123456"
    service_access_security_group     = "sg-123456"
    instance_profile                  = "arn:aws:iam::1234567890:instance-profile/EMR_EC2_DefaultRole"
  }
}

Modified config: change only the value of instance_profile

resource "aws_emr_cluster" "ad-hoc-query-cluster" {
  name                 = "ad-hoc-query-cluster"
  release_label        = "emr-5.12.0"
  applications         = ["Hadoop", "Hive", "HCatalog", "Spark"]
  log_uri              = "s3n://my-bucket/emr/logs/ad-hoc-query-cluster/"
  master_instance_type = "m3.xlarge"
  core_instance_type   = "m3.xlarge"
  core_instance_count  = 2
  service_role         = "EMR_DefaultRole"

  ec2_attributes {
    key_name                          = "my-pem-file"
    subnet_id                         = "subnet-123456"
    emr_managed_master_security_group = "sg-123456"
    emr_managed_slave_security_group  = "sg-123456"
    service_access_security_group     = "sg-123456"
    instance_profile                  = "arn:aws:iam::1234567890:instance-profile/Path/EMR_EC2_NewRole"
  }
}

Expected Behavior

Changes to any of the EC2 attributes must destroy the current cluster and create a new one.

Actual Behavior

Terraform reports the current cluster will be updated and report success doing so. Running the plan after apply will still report the resource changes.

Steps to Reproduce

1. Define an aws_emr_cluster resource
2. terraform plan will report the creation of the new cluster
3. terraform apply will create the cluster
4. Change the EC2 attribute instance_profile
5. terraform plan will report the cluster has changed and that it'll be updated
6. terraform apply will report the cluster has been changed
7. terraform plan will still report the previous change

$ > terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_emr_cluster.ad-hoc-query-cluster
      ec2_attributes.0.instance_profile: "arn:aws:iam::1234567890:instance-profile/EMR_EC2_DefaultRole" => "arn:aws:iam::1234567890:instance-profile/Path/EMR_EC2_NewRole"


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ > terraform apply
...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_emr_cluster.ad-hoc-query-cluster
      ec2_attributes.0.instance_profile: "arn:aws:iam::1234567890:instance-profile/EMR_EC2_DefaultRole" => "arn:aws:iam::1234567890:instance-profile/Path/EMR_EC2_NewRole"


Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_emr_cluster.ad-hoc-query-cluster: Modifying... (ID: j-ABCDEF123456)
  ec2_attributes.0.instance_profile: "arn:aws:iam::1234567890:instance-profile/EMR_EC2_DefaultRole" => "arn:aws:iam::1234567890:instance-profile/Path/EMR_EC2_NewRole"
aws_emr_cluster.ad-hoc-query-cluster: Modifications complete after 3s (ID: j-ABCDEF123456)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

$ > terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_emr_cluster.ad-hoc-query-cluster
      ec2_attributes.0.instance_profile: "arn:aws:iam::1234567890:instance-profile/EMR_EC2_DefaultRole" => "arn:aws:iam::1234567890:instance-profile/Path/EMR_EC2_NewRole"


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

[ddcprg]

I'll try to submit a patch today

[bflad]

The fix for this has been merged into master and will released with v1.15.0 of the AWS provider, likely tomorrow.

[bflad]

This has been released in version 1.15.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!