Updates of a vsphere_virtual_machine may always require the resource to be in a powered off state due to SetHardwareVersion

[arizvisa]

Terraform Version

Terraform v0.12.29

vSphere Provider Version

v1.21.1

Affected Resource(s)

I think it only affects this one:

• vsphere_virtual_machine

Terraform Configuration Files

Could look something like the following. The idea is that you deploy once, then modify "my_configuration" to point to a different datastore. This should perform an update.

locals {
  my_configuration = [
    {
      datastore = data.vsphere_datastore.ds1
    }
  ]
}

data "vsphere_datastore" "ds1" {
    name = "datastore-1"
    datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_datastore" "ds2" {
    name = "datastore-2"
    datacenter_id = data.vsphere_datacenter.datacenter.id
}

data "vsphere_virtual_machine" "template" {
  name = "/path/to/a/template"
  datacenter_id = data.vsphere_datacenter.datacenter.id
}

resource "vsphere_virtual_machine" "my-test-vm" {
  count = length(local.my_configuration)
  name = "my-test-vm.${count.index}"

  force_power_off = true                 # this is set to true by default
  datastore_id = local.my_configuration[count.index].datastore.id
  resource_pool_id = data.vsphere_compute_cluster.example-cluster.resource_pool_id
...
  clone {
    template_uuid = data.vsphere_virtual_machine.template.id
  }
...
}

Debug Output

Relevant logs follow, but they don't seem too useful. Relevant code is at vsphere/resource_vsphere_virtual_machine.go where resourceVSphereVirtualMachineUpdate is defined. It seems that the logic is explicitly checking what's being updated to set a "changed" flag, and then using that to determine whether to power off the machine. The datastore does not seem to be being checked.

2020-08-02T02:42:25.825-0500 [DEBUG] plugin.terraform-provider-vsphere: 2020/08/02 02:42:25 [DEBUG] Fetching properties for VM "/vsphere/vm/user/example/my-test-vm.0"
2020-08-02T02:42:25.921-0500 [DEBUG] plugin.terraform-provider-vsphere: 2020/08/02 02:42:25 [DEBUG] Fetching properties for VM "/vsphere/vm/user/example/my-test-vm.1"
2020-08-02T02:42:25.970-0500 [DEBUG] plugin.terraform-provider-vsphere: 2020/08/02 02:42:25 [DEBUG] Fetching properties for VM "/vsphere/vm/user/example/my-test-vm.2"
2020/08/02 02:42:26 [DEBUG] vsphere_virtual_machine.my-test-vm[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:26 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:26 [ERROR] <root>: eval: *terraform.EvalSequence, err: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:26 [DEBUG] vsphere_virtual_machine.my-test-vm[1]: apply errored, but we're indicating that via the Error pointer rather than returning it: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:26 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:26 [ERROR] <root>: eval: *terraform.EvalSequence, err: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:27 [DEBUG] vsphere_virtual_machine.my-test-vm[2]: apply errored, but we're indicating that via the Error pointer rather than returning it: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:27 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: The attempted operation cannot be performed in the current state (Powered on).
2020/08/02 02:42:27 [ERROR] <root>: eval: *terraform.EvalSequence, err: The attempted operation cannot be performed in the current state (Powered on).
2020-08-02T02:42:27.070-0500 [DEBUG] plugin: plugin process exited: path=/home/user/.terraform.d/plugins/terraform-provider-vsphere pid=108918
2020-08-02T02:42:27.070-0500 [DEBUG] plugin: plugin exited

Panic Output

No panic.

Expected Behavior

I expect it to move the machine to a different datastore, or at very least expose some way that I could influence the powerState for the vsphere_virtual_machine so that I can move it to a different datastore. It seems that govmomi exposes controlling power for machines with the PowerOn and PowerOff methods.

Actual Behavior

Instead of powering off the virtual machine according to the documentation which references failure during updating:

force_power_off - (Optional) If a guest shutdown failed or timed out while updating or destroying (see shutdown_wait_timeout), force the power-off of the virtual machine. Default: true. 

I get the following error. Is there some way I can shutdown the virtual machine with the "terraform-provider-vsphere" plugin?

Error: The attempted operation cannot be performed in the current state (Powered on).                                                                           
                                                                                                                                                                
  on main.tf line 39, in resource "vsphere_virtual_machine" "my-test-vm":                                                                                       
  39: resource "vsphere_virtual_machine" "my-test-vm" {                                                                                                         

Steps to Reproduce

1. We need to trigger an "update" step, so run terraform apply to first deploy a box.
2. After deployment, switch the datastore by modifying "my_configuration" if you're using my example (or some other means)
3. Run terraform apply, confirm that it's only making a change to the datastore, and then apply it.

Important Factoids

This is just for an internal tutorial I'm trying to write to promote usage of tf on our vSphere instance.

References

None.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[arizvisa]

So the issue seems to actually be here. Changing the hardware version requires the machine to be powered off. I'm pretty sure we'll need to explicitly check the version and compare if the update is actually changing the version instead of retaining it.

vsphere/resource_vsphere_virtual_machine.go:1666

	// Upgrade the VM's hardware version if needed.
	log.Printf("[DEBUG] the current vm properties are here: %#v", vprops)
	err = virtualmachine.SetHardwareVersion(vm, d.Get("hardware_version").(int))
	if err != nil {
		return nil, err
	}

[arizvisa]

I'm patching it right now and will let y'all know if this resolves it.

[arizvisa]

Okay, so what's happening is when SetHardwareVersion fetches the properties to compare the HwVersion if it changes, the properties it gets back has HwVersion set to the following.

Config:types.VirtualMachineConfigSummary{
    DynamicData:types.DynamicData{},
...
    HwVersion:""
},

Let me confirm with the docs on why the HwVersion is unset here.

[arizvisa]

Okay. So it seems that all our VMs have this field unset (both powered on and powered off). I reported this issue 4 days ago, and we updated from 6.5 to 6.7 on Tuesday so I'm not sure whether this is the same on versions earlier than 6.5.

Nonetheless, the JSON that vSphere gives back is the following. I'm not sure which HwVersion is being checked, but I imagine it's Summary.Config.

$ jq '.elements[].Object.Summary.Config' < fuck.json | grep HwVersion
  "HwVersion": ""

$ jq '.elements[].Object.Guest' < fuck.json | grep HwVersion
  "HwVersion": ""

$ jq '.elements[].Object.Summary.Guest' < fuck.json | grep HwVersion
  "HwVersion": ""

[arizvisa]

So I believe govmomi has these fields as optional (omitempty tag), but could be entirely mistaken here.

type VirtualMachineGuestSummary struct {
...
	HwVersion           string                    `xml:"hwVersion,omitempty"`
}

type VirtualMachineConfigSummary struct {
	HwVersion           string                       `xml:"hwVersion,omitempty"`
}

However, if you query the VirtualHardwareOption directly, then it guarantees that you have the version.

type VirtualHardwareOption struct {
	DynamicData

	HwVersion                int32                     `xml:"hwVersion"`
...

This corresponds to the docs for hwVersion (https://code.vmware.com/apis/196/vsphere/doc/vim.vm.VirtualHardwareOption.html#hwVersion), which is part of the ConfigOption structure (https://code.vmware.com/apis/196/vsphere/doc/vim.vm.ConfigOption.html#field_detail), which is explicitly queried with QueryConfigOption (https://code.vmware.com/apis/196/vsphere/doc/vim.EnvironmentBrowser.html#queryConfigOption).

So these attributes are using vm.Properties() which is using the default collector, and not explicitly calling QueryConfigOption to get these attributes. The summary configuration options don't seem to guarantee this field as being populated in the docs that I've seen, but the result from QueryConfigOption does.

[arizvisa]

Apparently a govc command that I haven't seen before called vm.option.info which I found by grepping for QueryConfigOption shows the hwversion:

$ govc vm.option.info -dump /fuzzfarm/vm/user/mybox
    Version:           "vmx-14",
    Description:       "ESXi 6.7 virtual machine",
    GuestOSDescriptor: []types.GuestOsDescriptor{
...
    HardwareOptions:     types.VirtualHardwareOption{
        HwVersion:           14,
        VirtualDeviceOption: []types.BaseVirtualDeviceOption{
            &types.VirtualSerialPortOption{
...

[arizvisa]

Updated title with a more general one as I believe this affects all updates of a vsphere_virtual_machine resource.

[arizvisa]

PR #1159 fixes this.

[tenthirtyam]

Hi @arizvisa - per your comment above, your PR #1159 resolved this issue.

Would you mind marking the issue as closed?

Many thanks,
Ryan

[arizvisa]

Yep. Got it.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.