Cannot update google_monitoring_alert_policy if related google_monitoring_notification_channel is deleted

[sergei-ivanov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.11

• provider.google v1.20.0

Affected Resource(s)

• google_monitoring_alert_policy
• google_monitoring_notification_channel

Terraform Configuration Files

# [INITIAL] terraform.tfvars file:
notification_email_addresses = [
  "[email protected]",
  "[email protected]",
  "[email protected]",
  "[email protected]", # This entry will be removed
]

# [MODIFIED] terraform.tfvars file:
notification_email_addresses = [
  "[email protected]",
  "[email protected]",
  "[email protected]",
]

variable "notification_email_addresses" {
  type        = "list"
  default     = []
}

resource "google_monitoring_notification_channel" "email" {
  count        = "${length(var.notification_email_addresses)}"
  project      = "${var.project}"
  enabled      = true
  display_name = "Send email to ${element(var.notification_email_addresses, count.index)}"
  type         = "email"

  labels = {
    email_address = "${element(var.notification_email_addresses, count.index)}"
  }
}

resource "google_monitoring_alert_policy" "disk-usage" {
  project      = "${var.project}"
  display_name = "Disk usage"
  enabled      = true
  combiner     = "OR"

  notification_channels = [
    "${google_monitoring_notification_channel.email.*.name}",
  ]

  conditions = [
    {
      display_name = "Free disk space on sda1"

      condition_threshold {
        filter = "metric.type=\"agent.googleapis.com/disk/percent_used\" resource.type=\"gce_instance\" metric.label.\"device\"=\"sda1\" metric.label.\"state\"=\"free\""

        duration        = "60s"
        comparison      = "COMPARISON_LT"
        threshold_value = "10"

        trigger {
          count = "1"
        }

        aggregations = [
          {
            alignment_period   = "60s"
            per_series_aligner = "ALIGN_MIN"
          },
        ]
      }
    },
  ]
}

Debug Output

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
  - destroy

Terraform will perform the following actions:

  ~ module.main.google_monitoring_alert_policy.disk-usage
      notification_channels.#: "5" => "4"
      notification_channels.4: "projects/dev/notificationChannels/6510479629081562055" => ""

  - module.main.google_monitoring_notification_channel.email[4]


Plan: 0 to add, 1 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.main.google_monitoring_notification_channel.email[4]: Destroying... (ID: projects/dev/notificationChannels/6510479629081562055)

Error: Error applying plan:

1 error(s) occurred:

* module.main.google_monitoring_notification_channel.email[4] (destroy): 1 error(s) occurred:

* google_monitoring_notification_channel.email.4: Error reading NotificationChannel: googleapi: Error 400: Could not delete "projects/dev/notificationChannels/6510479629081562055" because it is still being referenced by: "projects/dev/alertPolicies/9007997391602347955"

Panic Output

N/A

Expected Behavior

The list of email addresses is stored in a variable and is used to create a list of notification channels.
A monitoring policy is dependent on the list of notification channels in turn.
If an email is removed from the list, Terraform needs to update the policy before destroying the notification channel.

Actual Behavior

Terraform attempts to destroy notification channel before removing it from the associated policy. That fails with an error because the policy still refers to the notification channel.

Steps to Reproduce

1. terraform apply with the "INITIAL" version of terraform.tfvars
2. terraform apply with the "MODIFIED" version of terraform.tfvars

Important Factoids

N/A

References

N/A

[sergei-ivanov]

The only way to mitigate this at the moment is to remove the notification channel association from the monitoring policy manually:

$ gcloud alpha monitoring policies update --project=dev --remove-notification-channels=projects/dev/notificationChannels/6510479629081562055 projects/dev/alertPolicies/9007997391602347955

After that terraform plan can be reapplied.

[hrahuu]

This bug still exists in Terraform 0.12 too.

[edwardmedia]

I can repro it with v3.16.0

  # google_monitoring_notification_channel.email[3] will be destroyed
  - resource "google_monitoring_notification_channel" "email" {
      - display_name = "Send email to [email protected]" -> null
      - enabled      = true -> null
      - id           = "projects/myproject/notificationChannels/11974543105470211556" -> null
      - labels       = {
          - "email_address" = "[email protected]"
        } -> null
      - name         = "projects/myproject/notificationChannels/11974543105470211556" -> null
      - project      = "myproject" -> null
      - type         = "email" -> null
      - user_labels  = {} -> null
    }

---[ REQUEST ]---------------------------------------
DELETE /v3/projects/sunedward-1-autotest/notificationChannels/11974543105470211556?alt=json HTTP/1.1
Host: monitoring.googleapis.com

---[ RESPONSE ]--------------------------------------
 {
   "error": {
    "code": 400,
 "message": "Could not delete \"projects/myproject/notificationChannels/11974543105470211
556\" because it is still being referenced by: \"projects/myproject/alertPolicies/11564506291800433410\"",
    "status": "FAILED_PRECONDITION"
   }
 }

[rileykarson]

This is an issue in upstream Terraform, unfortunately. It decides the order that changes like this are made to resources and we can't do anything about it in the provider (which operates on a single-resource-at-a-time basis).

I don't know any canonical issue for this problem. If it remains a problem, I'd suggest filing a new one (and link back here!).

[sergei-ivanov]

One option might be to annotate notification_channels with ForceNew, so that whenever the list of channels changes, the whole google_monitoring_alert_policy is recreated.

A more generic issue on the Terraform side is probably this one:
hashicorp/terraform#8099

[rileykarson]

Yep, this statement from that issue describes the problem we're encountering:

Terraform knows that when it's doing any operation that affects both foo.bar and bar.foo it will always do the operation to foo.bar first.

That's based on this config:

resource "foo" "bar" {
    foobar = "${file("foobar")}"
}

resource "bar" "foo" {
    depends_on = ["foo.bar"]
}

We're seeing a similar behaviour here, where Terraform wants to modify the notification channel before the alert policy even though it needs to do it in the opposite order. ForceNew wouldn't help I don't think, because it would still try to process the notification channel first (and we can't add it anyways, since it would make adding channels destructive!).

In some cases, lifecycle.create_before_destroy works, but this isn't one of them unfortunately.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!