Needs clarification on the use of vsphere_virtual_disk

[javefang]

Hi there,

We are doing immutable infrastructure here with Terraform and VSphere. This means we are creating and destroying virtual machines a lot. During which we have a use-case which hopefully is very common to others as well.

For example, when we deploy a cluster of ETCD nodes. For each node we want to create a vsphere_virtual_machine and a separate vsphere_virtual_disk for storing persistent data. During an update, ideally we can keep the disk, recreate the VM and re-attach the data disk to the new VM. However, it is not very clear on if this is possible or how this should be done.

e.g. On the AWS provider, we can do this by creating an "aws_volume_attachment" to attach a given aws_ebs_volume to an aws_instance. On the VSphere provider I can't seem to find an equivalent.

Terraform Version

v0.10.2

Affected Resource(s)

Please list the resources as a list, for example:

• vsphere_virtural_machine
• vsphere_virtual_disk

What have I tried so far?

Initially we though we can just create a vsphere_virtual_disk at specific VMDK path, and create a "disk" section to reference the VMDK path when creating the vsphere_virtual_machine. (See configs below)

resource "vsphere_virtual_machine" "etcd" {
  name = "etcd-0"
  folder = "Terraform"
  cluster = "Dev"
  vcpu  = "2"
  memory = "4096"
  datacenter = "${var.vmware_datacenter}"
  domain = "example.com"
  dns_suffixes                   = ["${var.dns_domain}"]
  dns_servers                    = ["${split(",", var.dns_servers)}"]
  
  network_interface {
    label = "${var.vmware_network_label}"
  }
  
  disk {
    datastore = "${var.vmware_datastore}"
    template  = "Packer/etcd-${var.etcd_version}"
  }
  
  disk {
    datastore = "${var.vmware_datastore}"
    vmdk      = "${vsphere_virtual_disk.etcd_data.vmdk_path}"
  }
}

resource "vsphere_virtual_disk" "etcd_data" {
  size          = "${var.etcd_data_vol_size}"
  vmdk_path     = "${var.vmware_vmdk_path}/etcd-0-data.vmdk"
  datacenter    = "${var.vmware_datacenter}"
  datastore     = "${var.vmware_datastore}"
  adapter_type  = "lsiLogic"
  type          = "thin"
}

This approach has several problems:

1. Although "terraform plan" and "terraform apply" works for the first time. Running "terraform plan" again will attempting to attach the second disk again (and fail if you "terraform apply")
2. The terraform state for the VM does not show the second disk (which causes 1 above)
3. Destroying the VM only will delete the second disk as well.
4. Destroying the whole script will show failure when destroying the disk (as the disk is already destroyed when the VM was destroyed)

We then tried to create the data disk directly in the vsphere_virtual_machine section, and set the "keep_on_remove" option, hoping the disk can be kept and re-attached when the machine is recreated.

resource "vsphere_virtual_machine" "etcd" {
  name = "etcd-0"
  folder = "Terraform"
  cluster = "Dev"
  vcpu  = "2"
  memory = "4096"
  datacenter = "${var.vmware_datacenter}"
  domain = "example.com"
  dns_suffixes                   = ["${var.dns_domain}"]
  dns_servers                    = ["${split(",", var.dns_servers)}"]
  
  network_interface {
    label = "${var.vmware_network_label}"
  }
  
  disk {
    datastore = "${var.vmware_datastore}"
    template  = "Packer/etcd-${var.etcd_version}"
  }
  
  disk {
    datastore = "${var.vmware_datastore}"
    name = "etcd_data"
    size = "30"
    keep_on_remove = true
  }
}

When the VM is destroyed. The folder which contains VM files will remain, and only contains the data disk file (etcd_data.vmdk). Everything works correctly until now. However, when the VM is recreated with the same name. Instead of creating all the new VM files in the existing folder (etcd-0), except for the data disk file, a new folder is created (etcd-0-1), and a new data disk file is created and attached to the new VM...

It looks like we cannot use vsphere_virtual_disk to create persistent data disk for the VM. I wonder how others are using this resource?

How I think the problem could be resolved?

I feel like we should have a new resource "vsphere_virtual_disk_attachment", which use govmomi's api (see example of the kuberentes vsphere provider code) to control the attachment and detachment of disks created by vsphere_virtual_disk.

Example usage should be like

resource "vsphere_virtual_machine" "etcd" {
  name = "etcd-0"
  folder = "Terraform"
  cluster = "Dev"
  vcpu  = "2"
  memory = "4096"
  datacenter = "${var.vmware_datacenter}"
  domain = "example.com"
  dns_suffixes                   = ["${var.dns_domain}"]
  dns_servers                    = ["${split(",", var.dns_servers)}"]
  
  network_interface {
    label = "${var.vmware_network_label}"
  }
  
  disk {
    datastore = "${var.vmware_datastore}"
    template  = "Packer/etcd-${var.etcd_version}"
  }
}

resource "vsphere_virtual_disk" "etcd_data" {
  size          = "${var.etcd_data_vol_size}"
  vmdk_path     = "${var.vmware_vmdk_path}/etcd-0-data.vmdk"
  datacenter    = "${var.vmware_datacenter}"
  datastore     = "${var.vmware_datastore}"
  adapter_type  = "lsiLogic"
  type          = "thin"
}

resource "vsphere_virtual_disk_attachment" "etcd_data_attachment" {
  device_name = "/dev/sdb" # not sure if this is possible
  vmdk_path   = "${vsphere_virtual_disk.etcd_data.vmdk_path}"
  virtual_machine_name = "${vsphere_virtual_machine.etcd.name}"
}

And the behaviour will be

Create

1. Disk is created
2. VM is created and powered on
3. Disk is attached to the VM
   (if cloud-init is used, it must wait for the disk to attach before attempting to format and mount it)

Update (recreate the VM)

1. VM is powered down
2. Disk is detached
3. VM is destroyed
4. New VM is created and powered on
5. Disk is re-attached to the VM

Destroy

1. VM is powered down
2. Disk is detached
3. VM is destroyed
4. Disk is destroyed

I'm happy to contribute if someone could confirm if this is a genuine issue and if the proposal above makes sense?

Thanks

Xinghong

[vancluever]

Hey @javefang - this is a great solution to this problem! 👍

I currently have on my roadmap plans to revisit the vsphere_virtual_machine resource in general over the next few months as there are several current issues with the way it handles things like inactionable ephemeral guest information and device management (what this falls under and more), how it handles vMotion (or its lack of an ability to) and with how well it's equipped to handle workflows like deploying an original virtual machine, and deploying from OVF.

Hence, I want to hold off on any serious refactors of the resource until this work is properly scoped out as to avoid duplicating too much effort on a resource that could look significantly different after the work is complete.

The great part about this, however, is that by handing this responsibility off to another resource, we don't have to worry about pending refactors. 😄

I don't necessarily have this on my roadmap right now, but would gladly look over a PR and merge it if it were to come in!

Some comments I have on the PoC laid out:

device_name could possibly be omitted, or allowed to be selected as a general device class versus a explicit device name. Device node would not be possible as you have laid out here (and would not 100% translate too well into Windows land, of course). I'm thinking something along the lines of the adapter_type that's in the vsphere_virtual_disk resource. The resource could attach to an existing device of that class on the VM (or even better, this might be able to be specified somehow), or it could create an entirely different device of that class and add it to the VM. The latter approach might be better to help isolate the device from any configuration that's managed specifically in vsphere_virtual_machine.

Consider whether or not we should be making the disk independent by default or not - to ensure that it's excluded from snapshots. This is especially important for larger disks, and considering that, I would prefer that the default was independent.

Consider what is necessary to ensure that a VM does not delete the disk when the VM is deleted. It looks like you have a good consideration as to this workflow already. Actually, this workflow should be pretty natural, because tainting a VM resource should propagate down the graph, stopping at the attachment resource, meaning that the attachment resource will destroy itself before the VM destroys itself, ensuring that the disk is safely detached before the VM is deleted, which should be adequate to protect it from deletion.

Finally, I want to get into the habit of referencing VMs by their unique identifiers - either by MOID or UUID. The latter is what I think we are leaning on as this is what I've asked to have happen in #107.

---

Again, I'm not too sure when I'm personally going to be able to look into this, however I think this is a great step to making the state of the resource saner in general. If anyone takes this on and puts in a PR, please reference this issue!

[javefang]

That's very helpful thank you! I will try to pick up when I have a chance.

[qubitrenegade]

This is very much how the OpenStack provider works with "Creating" and attaching floating IPs to instances.

I think it makes a lot of sense to follow the same pattern as established in other providers.

Also, I think this model could be applied to interfaces, disks, and any additional "VM components".

Anyway, just wanted to throw my $0.02 in that @javefang resource seems elegant and would solve a lot of problems we've run into as far as wanting to customize a "base" or "minimal" template.

[witchbutter]

I am using terraform in a similar fashion in a development environment. I am having the same problems. I would add that there needs to be some means to handle raw device maps in the vsphere_virtual_disk resource. I also have issue with terraform refresh thinking it needs to reroll vms when a disk has been storage vmotioned from another location.

[alternico]

Would be a nice one to have.
Thanks

[eduncan911]

The documentation is lacking for "attached disks." I tried to start editing the docs, but digging around the docs and linking off to other repos, only to find several PRs pending for docs, I felt it better to share the solution here.

Note: There has been some additional work done in this area. And there now is an attach feature.

Note: my code below was created as a composable module, to where I source it in a parent module and set a bunch of vars. I have omitted the definitions of those vars but you can create your own.

Disk Attach Requirements

If defining a vsphere_virtual_disk resource and attaching it to a vsphere_virtual_machine resource like so:

provider "vsphere" {
  user                 = "${var.vsphere_user}"
  password             = "${var.vsphere_password}"
  vsphere_server       = "${var.vsphere_server}"
  allow_unverified_ssl = true
}

...snip...

resource "vsphere_virtual_disk" "data_volume" {
  count      = "${var.data_volume_enabled ? var.node_count : 0}"
  size       = "${var.data_volume_size}"
  vmdk_path  = "disks/${local.node_name_prefix}-data-${format("%02d", count.index+1)}.vmdk"
  datacenter = "${data.vsphere_datacenter.dc.name}"
  datastore  = "${data.vsphere_datastore.datastore.name}"
  type       = "eagerZeroedThick"

  lifecycle {
    prevent_destroy = true
  }
}

resource "vsphere_virtual_machine" "vm" {
  count            = "${var.node_count}"
  name             = "${local.node_name_prefix}-${element(random_id.node.*.hex, count.index)}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"

  ...snip...

  # must match cloned disks from your template, in order!
  disk {
    label        = "disk0"
    size         = "${data.vsphere_virtual_machine.template.disks.0.size}"
    unit_number  = 0
  }

  # then, you can attach additional disks setting unit_number uniquely
  disk {
    label        = "disk1"
    datastore_id = "${data.vsphere_datastore.datastore.id}"
    attach       = true
    path         = "${element(vsphere_virtual_disk.data_volume.*.vmdk_path, count.index)}"
    unit_number  = 1
  }
}

... then there a few undocumented requirements that are needed. I ran into these after validate, plan and apply were ran and everything was created and started - but yet I kept getting errors.

Running plan again after the failed apply will show you additional requirements!

The additional requirements are:

• attach is required and is a boolean! This boolean type was not clear in the docs and only showed the conversion error after a failed apply and then another plan. Set it to true.
• datastore_id is an undocumented! This only showed up after deploying the whole thing and running plan again, was I then told this was required. Set it to match your other datastore_id values.
• path is the physical path, which you can derive directly from your vsphere_virtual_disk resource. I highly recommend NOT duplicating the value here, and using the vmdk_path.
• unit_number is a documented requirement (though loosely commented as required for attach).

Afterwards, it's up to your Provisioning scripts to mount the disk, format it, etc.

Hopefully the docs get updated to help the next person along.

[rayterrill]

@eduncan911 does this process allow you to resize the disk?

[eduncan911]

@rayterrill does this process allow you to resize the disk?

If you are asking if resource "vsphere_virtual_disk" "data_volume" allows you to provision a disk, then change the size of that same resource, and resize... I have no idea. Never tried.

IMO, staying true to Terraform's descriptive nature, I'd say it should not. It should mark the resource as dirty and provision a new, larger/smaller disk.

But that's just my opinion. The vSphere Provider developers here may allow resizing. It's up to them and their code.

I'd say just try it. Provision a disk at 1 GB, change it to 2 GB, and apply again to see if it resizes or marks as dirty to replace.

[rayterrill]

@eduncan911 agree that the disk size piece needs to be better documented. Not sure I agree with the inability to resize disks - lots of resources allow modification vs just build a new. But I'll def give it a shot. Thanks for documenting your work! :)

[easyawslearn]

Does the issue still Open I am working with vSphere 6.7 where export give 2 files ( ovf and vmdk) and I want to create VM using ovf and vmdk. didn't find the way, can you please suggest the alternate for creating VM with ovf and vmdk file

[SamyCoenen]

After testing, resizing a vsphere_virtual_disk resource always seems to delete the disk.
Tested using Terraform v0.11.14
It's still the same behavior though as of today, just look at the source here

ForceNew: true, //TODO Can this be optional (resize)?
https://github.com/terraform-providers/terraform-provider-vsphere/blob/163f92b4259454e9f0c77dc2537c440b9fd38401/vsphere/resource_vsphere_virtual_disk.go#L41

[bill-rich]

It looks like we have the original request in this issue covered. #851 is open for dealing with the resizing issue. I'm going to close this one and we'll track it there.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!