Terraform doesn't delete the forwarding rules

[danishkamili]

Upon changing the VM variables the terraform update fails with the following error:

│ Error: Error when reading or editing TargetInstance: googleapi: Error 400: The target_instance resource 'ZONE:northamerica-northeast1-a/PROJECT:977287114090/TARGET_INSTANCE:myf5demo-ti-c389' is already being used by 'REGION:northamerica-northeast1/PROJECT:977287114090/FORWARDING_RULE:myf5demo-forwarding-rule-c389', resourceInUseByAnotherResource

Workaround: To manually delete the forwarding rules using the target instances. And, re-run terraform apply works.

[danishkamili]

terraform version
Terraform v1.2.8
on linux_amd64

• provider registry.terraform.io/hashicorp/google v4.53.1
• provider registry.terraform.io/hashicorp/random v3.4.3
• provider registry.terraform.io/hashicorp/time v0.9.1

[danishkamili]

Also, the VM's get destroyed and recreated for every modification to the files that doesn't change the VM config, for instance updating the forwarding rules also destroys the VM and recreated the same.

[JeffGiroux]

Which folder option did you deploy? standalone, HA LB, HA api?

If deploying HA via API and a failover already occurred or IP move occurred...hence updating the forwarding rule, then it will no longer match TFSTATE and result in a failure like you see. Just a guess without knowing the folder you deployed.

[danishkamili]

HA via API

[JeffGiroux]

enable debug silly logs for runtime init onboarding in order to see why VM onboard fails

Currently, f5 runtime init line looks like

/usr/local/bin/f5-bigip-runtime-init --config-file /config/cloud/runtime-init-conf.yaml 2>&1

You need to modify to look like...

export F5_BIGIP_RUNTIME_INIT_LOG_LEVEL=silly && /usr/local/bin/f5-bigip-runtime-init --config-file /config/cloud/runtime-init-conf.yaml 2>&1

Reference runtime init repo for more info on "Controls"
https://github.com/F5Networks/f5-bigip-runtime-init#controls

[AnthonyF5]

This occurred a few times for me on destroy, fixed with update to bigip.tf:

Forwarding rule for Public IP

resource "google_compute_forwarding_rule" "vip1" {
name = format("%s-forwarding-rule-%s", var.projectPrefix, random_id.buildSuffix.hex)
target = google_compute_target_instance.f5vm01.id
ip_address = google_compute_address.vip1.address
port_range = "1-65535"
depends_on = [ google_compute_target_instance.f5vm01 ]
}

[JeffGiroux]

@AnthonyF5 were you receiving the same error as original ticket issue? Same HA via API template?

[AnthonyF5]

Yes it was the same repo HA via API for GCP.

The second target became an issue on my last destroy so I updated depends_on to include both:
depends_on = [ google_compute_target_instance.f5vm01, google_compute_target_instance.f5vm02 ]

[JeffGiroux]

Next time this happens during a failed deploy, I would suggest trying the taint commands. This still seems like a tfstate issue due to the IP mapping and how instances are associated with target instances. In my testing, a failed or wiped VM needs to be tainted. Alternatively, terraform destroy the entire deployment and start over.

Suggestion

# Run the commands you need

terraform taint module.bigip.google_compute_instance.f5vm01
terraform taint module.bigip2.google_compute_instance.f5vm01
terraform taint google_compute_target_instance.f5vm01
terraform taint google_compute_target_instance.f5vm02
terraform taint google_compute_forwarding_rule.vip1