Can not delete ACM completely alongside beta-private-cluster

[elebioda]

TL;DR

ACM v21.0.0 can not delete cluster when ACM is bundled with cluster creation using a private cluster. Build a cluster with the new ACM module, worked. But when I attempt to delete the cluster I receive the error
Error: Error waiting for Deleting Membership: Error code 9, message: failed to uninstall Kubernetes resources for Membership.: failed to delete Membership resources: failed to delete resource: Internal error occurred: failed calling webhook "abandon-on-uninstall.cnrm.cloud.google.com": failed to call webhook: Post "https://abandon-on-uninstall.cnrm-system.svc:443/abandon-on-uninstall?timeout=3s": no endpoints available for service "abandon-on-uninstall"

Not sure if this is because the node_pool is deleted before the ACM module

Expected behavior

Can delete the Cluster and ACM module in one module

Observed behavior

No response

Terraform Configuration

data "google_project" "project" {
  project_id = var.project_id
}

# google_client_config and kubernetes provider must be explicitly specified like the following.
data "google_client_config" "default" {}

data "google_kms_key_ring" "keyring" {
  name     = var.kms_key_ring_name
  location = var.kms_key_ring_location
  project  = var.shared_vpc_project != "" ? var.shared_vpc_project : var.project_id
}

data "google_kms_crypto_key" "key" {
  name     = var.kms_crypto_key_name
  key_ring = data.google_kms_key_ring.keyring.id
}

module "gke" {
  source                     = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
  project_id                 = var.project_id
  name                       = var.cluster_name
  region                     = var.region
  network                    = var.vpc_name
  subnetwork                 = var.subnet_name
  network_project_id         = var.shared_vpc_project
  ip_range_pods              = var.pods_secondary_name
  ip_range_services          = var.svc_secondary_name
  firewall_inbound_ports     = ["9443", "15017", "15014", "8080"]
  datapath_provider          = "ADVANCED_DATAPATH"
  horizontal_pod_autoscaling = true
  network_policy             = false
  enable_private_endpoint    = false
  enable_private_nodes       = true
  grant_registry_access      = true
  master_ipv4_cidr_block     = var.master_ip_cidr
  istio                      = false
  cloudrun                   = false
  config_connector           = true
  remove_default_node_pool   = true

  node_pools = var.node_pools

  node_pools_oauth_scopes = {
    all = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_pools_labels = local.node_pool_labels

  notification_config_topic = google_pubsub_topic.updates.id

  cluster_resource_labels = {
    "asmv"    = "1-10-4-asm-6"
    "mesh_id" = "XXXX"
  }

  database_encryption = [
    {
      "key_name" : data.google_kms_crypto_key.key.id,
      "state" : "ENCRYPTED"
    }
  ]
}

data "google_secret_manager_secret_version" "ssh_key" {
  secret  = var.ssh_secret_name
  project = var.project_id
}

module "acm" {
  count             = var.acm_host ? 1 : 0
  source            = "../acm"
  project_id        = var.project_id
  cluster_name      = var.cluster_name
  cluster_location  = module.gke.location
  acm_git_repo      = var.acm_git_repo
  sync_branch       = var.sync_branch
  cluster_endpoint  = module.gke.endpoint
  ssh_key_file_path = "ssh.key"
  ssh_key_data      = data.google_secret_manager_secret_version.ssh_key.secret_data
  ssh_secret_name   = var.ssh_secret_name
}
resource "google_pubsub_topic" "updates" {
  name    = "cluster-updates-${var.cluster_name}"
  project = var.project_id
}

Terraform Version

1.1.9

Additional information

No response

[bharathkkb]

Thanks for the report @elebioda
I suspect the dependency was not resolved correctly in v20 due to #1189 which is now fixed in v21. I checked in our CI logs for this ACM example which seems to be destroying it in the correct order (ACM module first, followed by GKE module)

Could you retry with latest module version?

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days