provider/vsphere: network interface order changes

[thetuxkeeper]

Config:

  network_interface {
    label = "cld_tst1_access"
    ipv4_address = "10.30.8.240"
    ipv4_prefix_length = 23
    ipv4_gateway = "10.30.8.1"
  }

  network_interface {
    label = "cld_tst1_storage"
  }

Results in (removed ipv6 parts):

    network_interface.0.ip_address:         "" => "<computed>"
    network_interface.0.ipv4_address:       "10.30.0.36" => "10.30.8.240"
    network_interface.0.ipv4_gateway:       "" => "10.30.8.1" (forces new resource)
    network_interface.0.ipv4_prefix_length: "23" => "23"
    network_interface.0.label:              "cld_tst1_storage" => "cld_tst1_access" (forces new resource)
    network_interface.0.subnet_mask:        "" => "<computed>"
    network_interface.1.ip_address:         "" => "<computed>"
    network_interface.1.ipv4_address:       "10.30.8.240" => "<computed>"
    network_interface.1.ipv4_gateway:       "10.30.8.1" => ""
    network_interface.1.ipv4_prefix_length: "23" => "<computed>"
    network_interface.1.label:              "cld_tst1_access" => "cld_tst1_storage" (forces new resource)
    network_interface.1.subnet_mask:        "" => "<computed>"

As you can see, interface 0 in the config becomes interface 1 in vmware (confirmed over webinterface of vsphere API `<vsphere-server/mob/?moid=vm-&doPath=guest.net)

Perhaps a different approach for the index of the interfaces would be good (perhaps deviceConfigId?)?

[chrislovecnm]

i understand that this is a design issue, my question is how is it a problem? What is it breaking, how is it impacting us?

[thetuxkeeper]

TF would change/redeploy the affected VMs each time you trigger an apply (ForceNew for those parameters). Since there is no update implemented. The tests would also fail (non-empty plan) if they used multiple network interfaces.
With an update the issue wouldn't be as big as it is now. But still need a second apply to get it right (tests would also fail), if it wouldn't recreate the interfaces resulting in new vmware indices (which are possibly in a different order).

[chrislovecnm]

What about ignoring the order and keying on mac addresses? Does the interface have a vmware guid? Sounds like we need some api level advice as well.

Since there is no update implemented.

In TF or in govmomi?

But still need a second apply to get it right (tests would also fail), if it wouldn't recreate the interfaces resulting in new vmware indices (which are possibly in a different order).

This is something we need to get fixed or worked around better. Running a second apply will not fly.

[chrislovecnm]

@thetuxkeeper as you brought up we should move to using https://www.vmware.com/support/developer/vc-sdk/visdk25pubs/ReferenceGuide/vim.vm.device.VirtualDevice.html#key to identify, but I am wondering if we still need 0,1,2 type keys as well.

Specifically other projects like mantl need a list of devices. Can we do this non-breaking??

[chrislovecnm]

If not I can get the mantl code changed, and I don't think we need to worry about too many people integrating at this point. We need TF vsphere to run off of the keys, and not 'order'. This apples to all objects disks, CD-ROM, etc.

[thetuxkeeper]

@chrislovecnm : Thanks for confirming this.
A non-breaking change would be possible if there is a reliable way to match the existing interfaces with configured interfaces.
We have some interfaces in the config and some interfaces from the vmware API which we have to get together.
I couldn't find a vmware API call which returns the created device, so that we could save the ID after the creation of the device. Perhaps we could add something to the dynamicProperty Array in vmware (a generated ID). See https://www.vmware.com/support/developer/vc-sdk/visdk25pubs/ReferenceGuide/vim.vm.device.VirtualDeviceSpec.html => "Properties inherited from DynamicData". But I'm not sure if this is safe. (govmomi has no dynamicProperty in DynamicData)

[chrislovecnm]

@thetuxkeeper we need to engage the govmomi experts. What exactly do you need from the API?

[thetuxkeeper]

@chrislovecnm : I think we need to know how we can identify a device after creating the VM.
What I can think of right now:

• setting a known/generated ID when creating the device and matching it wherever we want a specific device
• or some API function that let's you create one device after another and returns the device object or the ID

[thetuxkeeper]

@chrislovecnm : Found some hints in govc code:

• how you find a device: https://github.com/vmware/govmomi/blob/6e991fc1ecadb031898ff6cd9cf53fe08f86116f/govc/vm/network/change.go#L82
• the implementation of the Name function: https://github.com/vmware/govmomi/blob/f3f81563e3ff8d783e038d000a7c9eadf7eb8032/object/virtual_device_list.go#L762

I'll try to implement something as starting point. There still the question how we get the ID/Name matched to the interfaces in the config.

[thetuxkeeper]

Implementation PR: #7154

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.