workload_metadata_config permadrift on google_container_node_pool resource

[wyardley]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.8
on darwin_amd64
+ provider registry.terraform.io/fluxcd/flux v0.3.1
+ provider registry.terraform.io/gavinbunney/kubectl v1.13.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/google v3.88.0
+ provider registry.terraform.io/hashicorp/google-beta v3.88.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.5.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/integrations/github v4.16.0

Affected Resource(s)

• .google_container_node_pool

Terraform Configuration Files

[not including the values of vars passed in from var file, but this should give an idea of how the module is being invoked)

module "gke" {
  source                       = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
  version                      = "v17.0.0" ## tried latest / pinning back to v16
  project_id                   = var.project_id
  name                         = var.kubernetes_cluster_name
  regional                     = true
  region                       = var.region
  zones                        = var.kubernetes_cluster_zone
  network                      = var.vpc_network_name
  create_service_account       = true
  dns_cache                    = true
  enable_private_endpoint      = false
  enable_private_nodes         = true
  master_ipv4_cidr_block       = "100.127.192.16/28"
  master_authorized_networks   = []
  master_global_access_enabled = false
  subnetwork                   = "xxx"
  ip_range_pods                = "xxx"
  ip_range_services            = "xxx"
  kubernetes_version           = "1.20.10-gke.1600"
  http_load_balancing          = true
  horizontal_pod_autoscaling   = true
  network_policy               = false
  remove_default_node_pool     = true
  initial_node_count           = 1
  gce_pd_csi_driver            = true
  node_pools                   = var.node_pools
  node_pools_oauth_scopes      = var.node_pools_oauth_scopes
  node_pools_labels            = var.node_pools_labels
  node_pools_taints            = var.node_pools_taints
  node_pools_tags              = var.node_pools_tags
}

Debug Output

2021-10-11T16:07:02.046-0700 [INFO]  provider.terraform-provider-google-beta_v3.88.0_x5: 2021/10/11 16:07:02 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-11T16:07:02.046-0700
2021-10-11T16:07:02.048-0700 [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/google-beta\"]" produced an unexpected new value for module.gke.google_container_node_pool.pools["default-node-pool"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .node_config[0].workload_metadata_config[0].mode: was null, but now cty.StringVal("GKE_METADATA")

I'm not going to put the full debug out in, but I notice that the response seems to have both fields set:

  "workloadMetadataConfig": {
   "nodeMetadata": "GKE_METADATA_SERVER",
   "mode": "GKE_METADATA"
  },

and the apply tries to PUT this:

PUT /v1beta1/projects/xxxxx/locations/us-west2/clusters/yyyy/nodePools/default-node-pool?alt=json&prettyPrint=false HTTP/1.1
Host: container.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.0.8 (+https://www.terraform.io) Terraform-Plugin-SDK/2.5.0 terraform-provider-google-beta/3.88.0 blueprints/terraform/terraform-google-kubernetes-engine:beta-private-cluster/v16.1.0
Content-Length: 99
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.16.2 gdcl/20210830
Accept-Encoding: gzip

{
 "nodePoolId": "default-node-pool",
 "workloadMetadataConfig": {
  "nodeMetadata": "GKE_METADATA_SERVER"
 }
}

Expected Behavior

Previously, no changes were shown with the same config (and even, recently, as far as I can tell, the same module / provider versions, though reverting back a couple of versions hasn't helped)

Actual Behavior

Getting the following permadrift on each run:

  # module.gke.google_container_node_pool.pools["default-node-pool"] will be updated in-place
  ~ resource "google_container_node_pool" "pools" {
        id                  = "xxx/nodePools/default-node-pool"
        name                = "default-node-pool"
[...]
          ~ workload_metadata_config {
              - mode          = "GKE_METADATA" -> null
                # (1 unchanged attribute hidden)
            }
            # (1 unchanged block hidden)

there is a deprecation warning from the module using the old attribute:

╷
│ Warning: Deprecated Attribute
│ 
│   with module.gke.google_container_node_pool.pools["default-node-pool"],
│   on .terraform/modules/gke/modules/beta-private-cluster/cluster.tf line 358, in resource "google_container_node_pool" "pools":
│  358:   node_config {
│ 
│ Deprecated in favor of mode.

Steps to Reproduce

1. terraform apply

Important Factoids

References

See some additional comments at terraform-google-modules/terraform-google-kubernetes-engine#1024

[rileykarson]

@megan07: We didn't think this would happen- weird! See GoogleCloudPlatform/magic-modules#5259 (comment)

[megan07]

I wonder if it's only for the specific GKE_METADATA_SERVER value, I didn't see it when I was testing with a different value, but was able to repro it with this.

[rileykarson]

That might do it. Unfortunate!

[wyardley]

Thanks for looking into it @megan07 @rileykarson!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.