permadrift on location_policy with beta_private_cluster

[wyardley]

TL;DR

With v4.41.0 of the terraform provider, we're seeing permadrift with autoscaling => location_policy (with module default settings for autoscaling). I was able to create a simple repro case below.

Expected behavior

Terraform to apply clean

Observed behavior

      ~ autoscaling {
          - location_policy      = "BALANCED" -> null
            # (4 unchanged attributes hidden)
        }

Terraform Configuration

terraform {
  required_version = "1.3.3"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.42.0"
    }
  }
}

variable "project" {
  type    = string
  default = "xyz"
}

variable "region" {
  type    = string
  default = "us-west2"
}

provider "google" {
  project               = var.project
  region                = var.region
  user_project_override = true
}

module "gke" {
  source                       = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
  version                      = "23.2.0"
  project_id                   = var.project
  name                         = "testcluster"
  regional                     = true
  region                       = var.region
  zones                        = ["us-west2-a", "us-west2-b"]
  network                      = "xxxx"
  create_service_account       = false
  dns_cache                    = true
  enable_private_endpoint      = false
  enable_private_nodes         = true
  master_ipv4_cidr_block       = "100.127.192.48/28"
  master_authorized_networks   = []
  master_global_access_enabled = false
  subnetwork                   = "xxx" # I'm guessing these could probably be defaults as well
  ip_range_pods                = "xxxx"
  ip_range_services            = "xxxx"
  kubernetes_version           = "1.24.3-gke.2100"
  remove_default_node_pool     = true
  initial_node_count           = 1
  gce_pd_csi_driver            = true

  node_pools = [{
    name               = "default-node-pool"
    machine_type       = "e2-standard-4"
    node_locations     = "us-west2-a,us-west2-b"
    min_count          = 1
    max_count          = 1
    node_metadata      = "GCE_METADATA"
    local_ssd_count    = 0
    disk_size_gb       = 100
    disk_type          = "pd-ssd"
    image_type         = "COS_CONTAINERD"
    auto_repair        = true
    auto_upgrade       = false
    enable_secure_boot = true
    preemptible        = false
  }]
}

Terraform Version

Terraform v1.3.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.42.0
+ provider registry.terraform.io/hashicorp/google-beta v4.42.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.14.0
+ provider registry.terraform.io/hashicorp/random v3.4.3

Additional information

Maybe has to do with deleting default nodepool, but cluster_autoscaling seems to have balanced mode as default. https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/latest/submodules/beta-private-cluster

I tried going back to 4.36, 4.38, 4.39 providers etc., and the problem persists, so maybe this is an API level change or something else?

If I run with TF_LOG=debug, I see

 "autoscaling": {
  "enabled": true,
  "minNodeCount": 1,
  "maxNodeCount": 1,
  "locationPolicy": "BALANCED"
 },

as well as

 "autoscaling": {
  "autoscalingProfile": "BALANCED"
 },

in API responses

Tested older and newer provider versions, but let me know if this seems to be an upstream provider (and / or Google API response changing) issue. I also tested jumping back to v22.1.0 of this module.

[HametAk]

I can report the same issue with https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/latest.

[bharathkkb]

Thanks for the report, I suspect this is an upstream API change since GoogleCloudPlatform/magic-modules#6370 was recently added.

[bharathkkb]

#1448 possible fix in progress

[mmn0o7]

Im also having problems with location_policy with private-cluster

  ~ resource "google_container_node_pool" "pools" {
        id                          = "projects/gcp-workcanvas-prod-1/locations/us-east1/clusters/gke-prod-us-east1/nodePools/default-node-pool"
        name                        = "default-node-pool"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          - location_policy      = "ANY" -> null
            # (4 unchanged attributes hidden)
        }

[bharathkkb]

#1452 now allows you to specify location_policy. We have not set a default yet as the feature is dependent on the kubernetes version but you can set location_policy instead to avoid the permadiff.

[wyardley]

Hi folks. Can we reopen? Still seeing this after the v24 release from #1447

[wyardley]

Oh I see, have to set it explicitly 😞

[mmn0o7]

does the fix work for node_pools under gke module or just for google_container_node_pool resource ?

[tback]

I also don't understand what to do regarding node_pools defined in a gke module. It seems there is nothing in the documentation of the module https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/24.0.0 .

[lauraseidler]

For node_pools, setting the location_policy key works for us.

[tback]

On pools nested in the gke module?

…

On Thu, 8 Dec 2022, 17:03 Laura Seidler, ***@***.***> wrote: For node_pools, setting the location_policy key works for us. — Reply to this email directly, view it on GitHub <#1445 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUL2A2QB53TSGW7RGH5L3WMIBEDANCNFSM6AAAAAARTSG7XA> . You are receiving this because you commented.Message ID: <terraform-google-modules/terraform-google-kubernetes-engine/issues/1445/1342949038 @github.com>