Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google_compute_instance_template.advanced_machine_features erroneously marked as computed #9436

Closed
karlkfi opened this issue Jun 24, 2021 · 11 comments · Fixed by GoogleCloudPlatform/magic-modules#5313, hashicorp/terraform-provider-google-beta#3786 or #10427
Assignees
Milestone

Comments

@karlkfi
Copy link

karlkfi commented Jun 24, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

terraform-provider-google v3.73.0

This version JUST added the advanced_machine_features field. So no other versions are affected yet.

Affected Resource(s)

  • google_compute_instance_template

Terraform Configuration Files

Example used to test KCC (does not specify AdvancedMachineFeatures, because it was only just added) :

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeInstanceTemplate
metadata:
  name: computeinstancetemplate-${uniqueId}
  labels:
    label-one: "value-one"
spec:
  description: a basic instance template example
  machineType: n1-standard-1
  region: us-central1
  metadata:
    - key: foo
      value: bar
    - key: bar
      value: baz
  disk:
    - sourceDiskRef:
        name: computedisk-${uniqueId}
      autoDelete: false
      boot: true
    - sourceImageRef:
        name: computeimage-${uniqueId}
      autoDelete: true
      boot: false
  networkInterface:
    - networkRef:
        name: computenetwork-${uniqueId}
      subnetworkRef:
        name: computesubnetwork-${uniqueId}

Error Output

This error is from the Config Connector integration tests after pulling in the new terraform-provider-google v3.73.0 code as dependency.

samples_test.go:297: timed out waiting for ready on ComputeInstanceTemplate with name 'srxlguaxsm', final status.conditions[0] status 'False' and reason 'UpdateFailed': Update call failed: cannot make changes to immutable field(s): [advancedMachineFeatures.#]

However, no changes were made to the advancedMachineFeatures field, because it's not even being specified by the existing integration tests (new field).

Expected Behavior

The test should pass, because the advancedMachineFeatures field is not being specified by the client nor changed.

Actual Behavior

The test fails, with the error: cannot make changes to immutable field(s): [advancedMachineFeatures.#].

Terraform Provider Google would probably always trigger an unnecessary re-create as long as advanced_machine_features is empty and an update was performed.

TODO: Validate the actual error behavior when using TF without KCC.

Probable Root Cause

The advanced_machine_features field is marked as Computed: true and ForceNew: true. So when calculating the diff, if the new value is empty, the diff sets RequiresNew: true, which triggers a ImmutableFieldsMutationError error from Config Connector, even if nothing changed.

https://github.com/hashicorp/terraform-plugin-sdk/blob/main/helper/schema/schema.go#L394

References

@karlkfi karlkfi added the bug label Jun 24, 2021
@edwardmedia edwardmedia self-assigned this Jun 25, 2021
@edwardmedia
Copy link
Contributor

@karlkfi tested the api behavior. advanced_machine_features is optional. If we don't set it in the config, then it will not be included in the request and the api will not send it back in the response which makes me think it should NOT impact the existing behavior. Can you think of any reasons that attempt to alter the value of advanced_machine_features in your case?

no changes were made to the advancedMachineFeatures field,

It is true, any changes needed for advanced_machine_features, it will trigger re-create the resource.

@rileykarson
Copy link
Collaborator

@edwardmedia: Grabbing this bug, since I'm onduty & have been talking w/ @karlkfi about it offline as well.

The error caused by KCC does not occur when using Terraform, I'll keep the bug open for a bit in case we decide to amend the behaviour of the field (although it's likely a breaking change to do so)

@karlkfi
Copy link
Author

karlkfi commented Jun 25, 2021

It is true, any changes needed for advanced_machine_features, it will trigger re-create the resource.

Yes, but I don't think this is desirable behavior.

Example 1

Removing values that aren't default should trigger re-create.

advanced_machine_features {
  threads_per_core = 1
  enable_nested_virtualization = true
}

to

// removed

This SHOULD cause an update/recreate, because it's not the default in the API and has explicit meaning. But it actually ignores the change.

$ terraform plan
...
No changes. Infrastructure is up-to-date.

Example 2

Removing the default values should be a noop.

advanced_machine_features {
  enable_nested_virtualization = false
}

to

// removed

This SHOULD be a noop, because it's equivalent to the default. But it's actually a noop because removal of advanced_machine_features is ignored. This is the desired behavior but not for the right reason.

$ terraform plan
...
No changes. Infrastructure is up-to-date.

Example 3

Adding the default values should be a noop.

// empty

to

advanced_machine_features {
  enable_nested_virtualization = false
}

This SHOULD be a noop, because it's equivalent to the default. But it triggers a recreate.

$ terraform plan
...
      + advanced_machine_features { # forces replacement
          + enable_nested_virtualization = false # forces replacement
        }
...
Plan: 1 to add, 0 to change, 1 to destroy.

Example 4

Adding the default values should be a noop.

// empty

to

advanced_machine_features {
  enable_nested_virtualization = false
  threads_per_core             = 0
}

This SHOULD be a noop, because it's equivalent to the default. But it triggers a recreate.

$ terraform plan
...
      + advanced_machine_features { # forces replacement
          + enable_nested_virtualization = false # forces replacement
          + threads_per_core             = 0 # forces replacement
        }
...
Plan: 1 to add, 0 to change, 1 to destroy.

Fix

Removing Computed:true should fix all of these issues.

I'm not sure why Computed was set on advanced_machine_features.

@karlkfi
Copy link
Author

karlkfi commented Jun 25, 2021

The tf i used for the above tests was just this. Change the project and you can test yourself.

provider "google" {
  region  = "us-central1"
  project = "cnrm-test-karl-1"
}

resource "google_compute_instance_template" "default" {
  name        = "example"
  machine_type         = "n1-standard-1"

  disk {
    source_image      = "debian-cloud/debian-9"
    auto_delete       = false
    boot              = true
  }

  network_interface {
    network = "default"
  }
}

@upodroid
Copy link
Contributor

Karl, I initially tried those permutations for google_compute_instance in GoogleCloudPlatform/magic-modules#4849.

Has that bug in 4849 been fixed yet?

@karlkfi
Copy link
Author

karlkfi commented Jun 25, 2021

GoogleCloudPlatform/magic-modules#4849 looks like it also specifies Computed: true on the advanced_machine_features field.

I suggest changing it and adding more tests to validate the behavior works as desired for these use cases.

@karlkfi
Copy link
Author

karlkfi commented Jun 25, 2021

Oh, I see. You reported a bug in the docs/api not agreeing about the field being mutable. I don't have an snwer for that, but I can try to follow up.

The issue in this ticket is about unexpected behavior in the TF provider from your impl in #9363

Example 1 is the worst offender. It ignores a change when it should cause an update/recreate.

The other examples here are minor issues for TF users, because the template itself is immutable, but have caused another issue in Config Connector, which depends on the TF provider code.

@rileykarson
Copy link
Collaborator

Speaking about changing the instance template schema: Even though the impact is likely to be minor, removing the Computed: true, value from the field(s) in instance template will be a breaking change, as they'll cause a diff (and in fact recreate the resource) when removed instead of performing a noop. Based on that, I've triaged it as a breaking change.

@karlkfi: Those are all well-known behaviours of marking a field Optional and Computed, and the implications of the setting were understood by the reviewer of the change. They are not worth testing specifically on every change that introduces an Optional and Computed field. Additionally, the level of testing on #9363 was deemed sufficient by a maintainer of this repository. If they felt there was insufficient testing, they would have asked for more.

Note that Config Connector uses a private (unsupported) interface to interact with the provider, and that issues for Config Connector are not always issues for Terraform, such as this case. The configuration of the field(s) in #9363 are valid for a Terraform field and Config Connector should handle them correctly. In fact, it's entirely possible that there is a configuration of the field that requires Computed: true to be set as GCE can return values for blocks in unexpected circumstances!

@karlkfi
Copy link
Author

karlkfi commented Jun 28, 2021

Do we have an example where any of these fields are computed by the API, for either instances or instance templates?

Checking the responses from the API (using gcloud --log-http), it doesn't look like it.

I didn't test every permutation, but here are a few examples. I got the same results from both instances and instance templates.

Below, we see that for instances, advancedMachineFeatures is not computed when neither threadsPerCore or enableNestedVirtualization were specified by the user.

$ gcloud beta compute instances describe example --zone us-central1-b --log-http
...
{
  "id": "<redacted>",
  "creationTimestamp": "2021-06-28T10:55:32.735-07:00",
  "name": "example",
  "tags": {
    "fingerprint": "<redacted>"
  },
  "machineType": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/machineTypes/n1-standard-1",
  "status": "RUNNING",
  "zone": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/global/networks/default",
      "subnetwork": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/regions/us-central1/subnetworks/default",
      "networkIP": "10.128.0.10",
      "name": "nic0",
      "fingerprint": "<redacted>",
      "kind": "compute#networkInterface"
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "mode": "READ_WRITE",
      "source": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/disks/example",
      "deviceName": "persistent-disk-0",
      "index": 0,
      "boot": true,
      "autoDelete": true,
      "licenses": [
        "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/licenses/debian-9-stretch"
      ],
      "interface": "SCSI",
      "guestOsFeatures": [
        {
          "type": "VIRTIO_SCSI_MULTIQUEUE"
        }
      ],
      "diskSizeGb": "10",
      "kind": "compute#attachedDisk"
    }
  ],
  "metadata": {
    "fingerprint": "<redacted>",
    "kind": "compute#metadata"
  },
  "selfLink": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/instances/example",
  "scheduling": {
    "onHostMaintenance": "MIGRATE",
    "automaticRestart": true,
    "preemptible": false
  },
  "cpuPlatform": "Intel Haswell",
  "labelFingerprint": "<redacted>",
  "startRestricted": false,
  "deletionProtection": false,
  "fingerprint": "<redacted>",
  "lastStartTimestamp": "2021-06-28T10:55:41.453-07:00",
  "kind": "compute#instance"
}

Below, we see that for instance templates, advancedMachineFeatures is not computed when neither threadsPerCore or enableNestedVirtualization were specified by the user.

$ gcloud compute instance-templates describe example --log-http
...
{
  "id": "<redacted>",
  "creationTimestamp": "2021-06-28T10:54:15.663-07:00",
  "name": "example",
  "description": "",
  "properties": {
    "machineType": "n1-standard-1",
    "networkInterfaces": [
      {
        "network": "https://www.googleapis.com/compute/v1/projects/cnrm-test-karl-1/global/networks/default",
        "name": "nic0",
        "kind": "compute#networkInterface"
      }
    ],
    "disks": [
      {
        "type": "PERSISTENT",
        "mode": "READ_WRITE",
        "deviceName": "persistent-disk-0",
        "index": 0,
        "boot": true,
        "initializeParams": {
          "sourceImage": "projects/debian-cloud/global/images/family/debian-9",
          "diskType": "pd-standard"
        },
        "interface": "SCSI",
        "kind": "compute#attachedDisk"
      }
    ],
    "metadata": {
      "fingerprint": "<redacted>",
      "kind": "compute#metadata"
    },
    "scheduling": {
      "onHostMaintenance": "MIGRATE",
      "automaticRestart": true,
      "preemptible": false
    }
  },
  "selfLink": "https://www.googleapis.com/compute/v1/projects/cnrm-test-karl-1/global/instanceTemplates/example",
  "kind": "compute#instanceTemplate"
}

Below, we see that for instances threadsPerCore is returned when specified, and enableNestedVirtualization is not returned when not specified.

$ gcloud beta compute instances create example --zone us-central1-b --machine-type n1-standard-2 --threads-per-core 1

$ gcloud beta compute instances describe example --zone us-central1-b --log-http
...
{
  "id": "<redacted>",
  "creationTimestamp": "2021-06-28T11:26:51.972-07:00",
  "name": "example",
  "tags": {
    "fingerprint": "<redacted>"
  },
  "machineType": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/machineTypes/n1-standard-2",
  "status": "RUNNING",
  "zone": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/global/networks/default",
      "subnetwork": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/regions/us-central1/subnetworks/default",
      "networkIP": "10.128.0.11",
      "name": "nic0",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "natIP": "146.148.111.157",
          "networkTier": "PREMIUM",
          "kind": "compute#accessConfig"
        }
      ],
      "fingerprint": "<redacted>",
      "kind": "compute#networkInterface"
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "mode": "READ_WRITE",
      "source": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/disks/example",
      "deviceName": "persistent-disk-0",
      "index": 0,
      "boot": true,
      "autoDelete": true,
      "licenses": [
        "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/licenses/debian-10-buster"
      ],
      "interface": "SCSI",
      "guestOsFeatures": [
        {
          "type": "UEFI_COMPATIBLE"
        },
        {
          "type": "VIRTIO_SCSI_MULTIQUEUE"
        }
      ],
      "diskSizeGb": "10",
      "kind": "compute#attachedDisk"
    }
  ],
  "metadata": {
    "fingerprint": "<redacted>",
    "kind": "compute#metadata"
  },
  "serviceAccounts": [
    {
      "email": "[email protected]",
      "scopes": [
        "https://www.googleapis.com/auth/devstorage.read_only",
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring.write",
        "https://www.googleapis.com/auth/pubsub",
        "https://www.googleapis.com/auth/service.management.readonly",
        "https://www.googleapis.com/auth/servicecontrol",
        "https://www.googleapis.com/auth/trace.append"
      ]
    }
  ],
  "selfLink": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/instances/example",
  "scheduling": {
    "onHostMaintenance": "MIGRATE",
    "automaticRestart": true,
    "preemptible": false
  },
  "cpuPlatform": "Intel Haswell",
  "labelFingerprint": "<redacted>",
  "startRestricted": false,
  "deletionProtection": false,
  "shieldedVmConfig": {
    "enableSecureBoot": false,
    "enableVtpm": true,
    "enableIntegrityMonitoring": true
  },
  "shieldedVmIntegrityPolicy": {
    "updateAutoLearnPolicy": true
  },
  "shieldedInstanceConfig": {
    "enableSecureBoot": false,
    "enableVtpm": true,
    "enableIntegrityMonitoring": true
  },
  "shieldedInstanceIntegrityPolicy": {
    "updateAutoLearnPolicy": true
  },
  "fingerprint": "<redacted>",
  "advancedMachineFeatures": {
    "threadsPerCore": 1
  },
  "lastStartTimestamp": "2021-06-28T11:26:59.918-07:00",
  "kind": "compute#instance"
}

Below, we see that for instance enableNestedVirtualization is returned when specified, and threadsPerCore is not returned when not specified.

$ gcloud beta compute instances create example --zone us-central1-b --machine-type n1-standard-2 --enable-nested-virtualization --min-cpu-platform "Intel Haswell"

$ gcloud beta compute instances create example --zone us-central1-b --machine-type n1-standard-2 --enable-nested-virtualization --min-cpu-platform "Intel Haswell"
...
{
  "id": "<redacted>",
  "creationTimestamp": "2021-06-28T11:31:39.961-07:00",
  "name": "example",
  "tags": {
    "fingerprint": "<redacted>"
  },
  "machineType": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/machineTypes/n1-standard-2",
  "status": "RUNNING",
  "zone": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/global/networks/default",
      "subnetwork": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/regions/us-central1/subnetworks/default",
      "networkIP": "10.128.0.12",
      "name": "nic0",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "natIP": "35.222.97.143",
          "networkTier": "PREMIUM",
          "kind": "compute#accessConfig"
        }
      ],
      "fingerprint": "<redacted>",
      "kind": "compute#networkInterface"
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "mode": "READ_WRITE",
      "source": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/disks/example",
      "deviceName": "persistent-disk-0",
      "index": 0,
      "boot": true,
      "autoDelete": true,
      "licenses": [
        "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/licenses/debian-10-buster"
      ],
      "interface": "SCSI",
      "guestOsFeatures": [
        {
          "type": "UEFI_COMPATIBLE"
        },
        {
          "type": "VIRTIO_SCSI_MULTIQUEUE"
        }
      ],
      "diskSizeGb": "10",
      "kind": "compute#attachedDisk"
    }
  ],
  "metadata": {
    "fingerprint": "<redacted>",
    "kind": "compute#metadata"
  },
  "serviceAccounts": [
    {
      "email": "[email protected]",
      "scopes": [
        "https://www.googleapis.com/auth/devstorage.read_only",
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring.write",
        "https://www.googleapis.com/auth/pubsub",
        "https://www.googleapis.com/auth/service.management.readonly",
        "https://www.googleapis.com/auth/servicecontrol",
        "https://www.googleapis.com/auth/trace.append"
      ]
    }
  ],
  "selfLink": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/zones/us-central1-b/instances/example",
  "scheduling": {
    "onHostMaintenance": "MIGRATE",
    "automaticRestart": true,
    "preemptible": false
  },
  "cpuPlatform": "Intel Haswell",
  "labelFingerprint": "<redacted>",
  "minCpuPlatform": "Intel Haswell",
  "startRestricted": false,
  "deletionProtection": false,
  "shieldedVmConfig": {
    "enableSecureBoot": false,
    "enableVtpm": true,
    "enableIntegrityMonitoring": true
  },
  "shieldedVmIntegrityPolicy": {
    "updateAutoLearnPolicy": true
  },
  "shieldedInstanceConfig": {
    "enableSecureBoot": false,
    "enableVtpm": true,
    "enableIntegrityMonitoring": true
  },
  "shieldedInstanceIntegrityPolicy": {
    "updateAutoLearnPolicy": true
  },
  "fingerprint": "<redacted>",
  "advancedMachineFeatures": {
    "enableNestedVirtualization": true
  },
  "lastStartTimestamp": "2021-06-28T11:31:51.744-07:00",
  "kind": "compute#instance"
}

Below, we see that for instance templates enableNestedVirtualization is returned when specified, and threadsPerCore is not returned when not specified.

$ gcloud beta compute instance-templates create example --machine-type n1-standard-2 --enable-nested-virtualization --min-cpu-platform "Intel Haswell"

$ gcloud beta compute instance-templates describe example --log-http
...
{
  "id": "<redacted>",
  "creationTimestamp": "2021-06-28T11:44:41.308-07:00",
  "name": "example",
  "description": "",
  "properties": {
    "machineType": "n1-standard-2",
    "canIpForward": false,
    "networkInterfaces": [
      {
        "network": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/global/networks/default",
        "name": "nic0",
        "accessConfigs": [
          {
            "type": "ONE_TO_ONE_NAT",
            "name": "external-nat",
            "networkTier": "PREMIUM",
            "kind": "compute#accessConfig"
          }
        ],
        "kind": "compute#networkInterface"
      }
    ],
    "disks": [
      {
        "type": "PERSISTENT",
        "mode": "READ_WRITE",
        "deviceName": "persistent-disk-0",
        "index": 0,
        "boot": true,
        "initializeParams": {
          "sourceImage": "https://compute.googleapis.com/compute/beta/projects/debian-cloud/global/images/family/debian-10"
        },
        "autoDelete": true,
        "kind": "compute#attachedDisk"
      }
    ],
    "metadata": {
      "fingerprint": "<redacted>",
      "kind": "compute#metadata"
    },
    "serviceAccounts": [
      {
        "email": "default",
        "scopes": [
          "https://www.googleapis.com/auth/devstorage.read_only",
          "https://www.googleapis.com/auth/logging.write",
          "https://www.googleapis.com/auth/monitoring.write",
          "https://www.googleapis.com/auth/pubsub",
          "https://www.googleapis.com/auth/service.management.readonly",
          "https://www.googleapis.com/auth/servicecontrol",
          "https://www.googleapis.com/auth/trace.append"
        ]
      }
    ],
    "scheduling": {
      "onHostMaintenance": "MIGRATE",
      "automaticRestart": true,
      "preemptible": false
    },
    "minCpuPlatform": "Intel Haswell",
    "advancedMachineFeatures": {
      "enableNestedVirtualization": true
    }
  },
  "selfLink": "https://www.googleapis.com/compute/beta/projects/cnrm-test-karl-1/global/instanceTemplates/example",
  "kind": "compute#instanceTemplate"
}

@rileykarson
Copy link
Collaborator

@slevenick: marking 4.0.0-staged given GoogleCloudPlatform/magic-modules#5313 is approved.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 27, 2021
@github-actions github-actions bot added forward/review In review; remove label to forward service/compute-instances labels Jan 14, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.