Reference Eventarc created subscription to change its configuration

[fedexist]

Hi, I have an Eventarc trigger which creates an underlying topic and related subscription. Thing is, I'd like to update the subscription configuration by changing its default value of ack_deadline_seconds.

I tried creating the following resource

resource "google_pubsub_subscription" "eventarc-sub" {
  project = var.project_id
  provider = google
  name    = element(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].subscription), length(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].subscription)) - 1)
  topic   = element(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].topic), length(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].topic)) - 1)

  ack_deadline_seconds = 600

  depends_on = [
    google_eventarc_trigger.trigge_name
  ]
}

but it fails with the error

Error: Error creating Subscription: googleapi: Error 409: Resource already exists in the project

So, apparently, the resources created by Eventarc are not contained by Terraform state? Is it possible to do what I want? I certainly do not want to manually update the subscription

What is an alternative?
Run apply (the tf module without the configuration change) -> import (of the created subscription, got from the resource outputs) -> apply (the configuration change) ?

Maybe it should be possible to set a subscription to the eventarc_trigger resource? Just like it's possible to set an existing pubsub topic.

[edwardmedia]

@fedexist I don't know why you received Error 409: Resource already exists in the project. What resource(s)? Mind share the full debug log so I could have a big picture. You have mentioned other attributes, mind share how you config other resources? I have not exactly followed what you described. Better we can have the detailed configs & steps, etc....

[fedexist]

Sure thing, I'll try to explain myself better.

This would be the complete configuration and should be enough to reproduce the issue:

resource "google_eventarc_trigger" "trigger_name" {
  name     = "trigger_name"
  location = "eu"
  project  = var.project_id
  provider = google
  matching_criteria {
    attribute = "type"
    value     = "google.cloud.storage.object.v1.finalized"
  }

  matching_criteria {
    attribute = "bucket"
    value     = var.bucket_name
  }

  destination {
    cloud_run_service {
      service = "cloud-run-to-be-triggered"
      region  = var.region
      path    = "/v1/ingest"
    }
  }
  service_account = var.service_account_email
  labels          = local.labels
}

resource "google_pubsub_subscription" "eventarc-sub" {
  project = var.project_id
  provider = google
  name    = element(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].subscription), length(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].subscription)) - 1)
  topic   = element(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].topic), length(split("/", google_eventarc_trigger.trigger_name.transport[0].pubsub[0].topic)) - 1)

  ack_deadline_seconds = 600

  depends_on = [
    google_eventarc_trigger.trigger_name
  ]
}

What I think is happening is this:

1. The Eventarc trigger gets created by Terraform -> this creates a PubSub topic and a subscription, resources "managed" by the Eventarc trigger
2. I retrieve the created subscription from the resource outputs
3. I use that value in the google_pubsub_subscription resource to update the ack_deadline_seconds attribute to 600
4. It fails with 409 Error because terraform is trying to create a new subscription with the same name, and not updating the existing one

Terraform will perform the following actions:

  # google_eventarc_trigger.ania_ingestion_trigger will be created
  + resource "google_eventarc_trigger" "trigger_name" {
      + create_time     = (known after apply)
      + etag            = (known after apply)
      + id              = (known after apply)

      + location        = "eu"
      + name            = "trigger_name"
      + project         = "it-nonprod-gen-advana-000031"
      + service_account = "sa-ania-ibd-processor@it-nonprod-gen-advana-000031.iam.gserviceaccount.com"
      + uid             = (known after apply)
      + update_time     = (known after apply)

      + destination {
          + cloud_run_service {
              + path    = "gimo/v1/ingest"
              + region  = "europe-west3"
              + service = "cloud-run-to-be-triggered"
            }
        }

      + matching_criteria {
          + attribute = "bucket"
          + value     = "bucket-name"
        }
      + matching_criteria {
          + attribute = "type"
          + value     = "google.cloud.storage.object.v1.finalized"
        }

      + transport {
          + pubsub {
              + subscription = (known after apply)
              + topic        = (known after apply)
            }
        }
    }

  # google_pubsub_subscription.eventarc-ingestion-sub will be created
  + resource "google_pubsub_subscription" "eventarc-sub" {
      + ack_deadline_seconds       = 600
      + id                         = (known after apply)
      + message_retention_duration = "604800s"
      + name                       = (known after apply)
      + project                    = "it-nonprod-gen-advana-000031"
      + topic                      = (known after apply)

      + expiration_policy {
          + ttl = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: terraform.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "terraform.plan"

+ pwd
+ terraform '-chdir=/var/lib/jenkins/workspace/sw00250-aa-sources-gi/ania-ibd-processor/terraform-ania-ibd-processor-dev/tf' apply -no-color -compact-warnings terraform.plan
google_eventarc_trigger.ania_ingestion_trigger: Creating...
google_eventarc_trigger.ania_ingestion_trigger: Still creating... [10s elapsed]
google_eventarc_trigger.ania_ingestion_trigger: Creation complete after 16s [id=projects/it-nonprod-gen-advana-000031/locations/eu/triggers/ania-ingestion-trigger]
google_pubsub_subscription.eventarc-ingestion-sub: Creating...

Error: Error creating Subscription: googleapi: Error 409: Resource already exists in the project (resource=eventarc-eu-trigger-name-sub-624).

  with google_pubsub_subscription.eventarc-ingestion-sub,
  on gen_data_red.tf line 126, in resource "google_pubsub_subscription" "eventarc-ingestion-sub":
 126: resource "google_pubsub_subscription" "eventarc-ingestion-sub" {

Finished: FAILURE

[edwardmedia]

@fedexist Now I see. Instead of using google_pubsub_subscription to CREATE a new resource, you may import the existing google_pubsub_subscription first and then update its attributes on the existing one. Does this make sense?

[fedexist]

@edwardmedia Yep, it makes sense and that's what I was asking in the first place.

Still, I was wondering if it wouldn't make more sense that this resource gets automatically imported or, even, add the possibility to configure these resources directly from the eventarc trigger resource.

I don't know if it can be considered a feature request or something like that, but it would be nice to add that to the eventarc_trigger resource.

[edwardmedia]

@fedexist I don't believe this could happen: this resource gets automatically imported. Terraform does not have knowledge about what other changes are made because API usually does not send the other underlying changes back. Do you have other questions?

[fedexist]

@edwardmedia I see your point, but, still, when the trigger is created it has both the topic and the subscription created, available in google_eventarc_trigger.trigger_name.transport[0].pubsub[0].subscription and google_eventarc_trigger.trigger_name.transport[0].pubsub[0].topic, so the information about the created resources is there, it's just not being inserted in the tf state as an additional resource, and I feel like it could belong in it.

Anyway, thanks your help, integrating the import of an automatically created resource in our current pipeline run by CI/CD might be a bit tricky and certainly a solution with a single apply would be much better.

[edwardmedia]

@fedexist thanks for your point and understand what you expected, but again as I explained earlier it is impossible to implement that at the provider level. Closing the question now

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.