aws_s3_bucket_notifications can be silently overwritten

[zestysoft]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.0
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v3.69.0

Also happens with Terraform v0.14.7 and aws provider v3.68.0

Affected Resource(s)

• aws_s3_bucket_notification

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
  required_providers {
    aws = "~> 3.69.0"
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "bucket" {
  bucket = "test-bucket"
}

resource "aws_sqs_queue" "queue" {
  name = "s3-event-notification-queue"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:*:*:s3-event-notification-queue",
      "Condition": {
        "ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.bucket.arn}" }
      }
    }
  ]
}
POLICY
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.bucket.id

  queue {
    id            = "image-upload-event"
    queue_arn     = aws_sqs_queue.queue.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "images/"
  }

  queue {
    id            = "video-upload-event"
    queue_arn     = aws_sqs_queue.queue.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "videos/"
  }
}

#resource "aws_s3_bucket_notification" "bucket_notification2" {
#  bucket = aws_s3_bucket.bucket.id
#
#  queue {
#    id            = "audio-upload-event"
#    queue_arn     = aws_sqs_queue.queue.arn
#    events        = ["s3:ObjectCreated:*"]
#    filter_prefix = "audio/"
#  }
#}

Expected Behavior

Add the queue to the existing list of queues
or
Warn that applying will change / overwrite what's existing

Actual Behavior

With TF v0.14.7, after previously applying the configuration file above, uncommenting the second aws_s3_bucket_notification block and running terraform plan only shows new code will be added to AWS without any notice that existing queues will be replaced:

# aws_s3_bucket_notification.bucket_notification2 will be created
  + resource "aws_s3_bucket_notification" "bucket_notification2" {
      + bucket = "test-bucket"
      + id     = (known after apply)

      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "videos/"
          + id            = "audio-upload-event"
          + queue_arn     = "arn:aws:sqs:us-east-1:xxx:s3-event-notification-queue"
        }
    }

The same thing happens with v1.1.0, but includes an incorrect notice that objects have changed outside of Terraform:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_s3_bucket.bucket has changed
  ~ resource "aws_s3_bucket" "bucket" {
        id                          = "test-bucket"
      + tags                        = {}
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # aws_sqs_queue.queue has changed
  ~ resource "aws_sqs_queue" "queue" {
        id                                = "https://sqs.us-east-1.amazonaws.com/xxx/s3-event-notification-queue"
        name                              = "s3-event-notification-queue"
      + tags                              = {}
        # (13 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes
using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket_notification.bucket_notification2 will be created
  + resource "aws_s3_bucket_notification" "bucket_notification2" {
      + bucket = "test-bucket"
      + id     = (known after apply)

      + queue {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "audio/"
          + id            = "audio-upload-event"
          + queue_arn     = "arn:aws:sqs:us-east-1:xxx:s3-event-notification-queue"
        }
    }

Looking at the s3 bucket's details in the AWS console after applying, only shows the single audio queue:

Steps to Reproduce

1. Apply the terraform as is with the comment block intact.
2. Uncomment the block of code at the bottom and apply again.

[ewbankkit]

Hi @zestysoft 👋 Thank you for submitting this and this is an excellent use case of somewhere that Terraform and the Terraform AWS Provider could be much more helpful since in many cases they have enough information to return an error upfront during planning instead of unexpected behavior during apply.

I believe this falls under the provider-wide enhancement proposal of #14394, so by adding this link here it will add a reference to that issue so we can include it as a use case when thinking about the implementation details. Since this is likely something we will want more broadly across many resources, I'm going to close this particular issue to consolidate discussions, efforts, and prioritization on the topic while the reference would serve as the cue to make this specific resource one of the initial implementations. I would suggest those 👍 upvoting and subscribing here to do so on #14394 so we can appropriately gauge interest. Please feel free to provide feedback there.

[ewbankkit]

Relates #501.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.