Terraform and Azure: no way to control NIC ordering on a VM

[vinayvenkat]

Terraform Version

Terraform v0.9.5

Terraform Configuration Files

resource "azurerm_network_interface" "VNIC0" {
  name                = "${join("", list("FW", "web-nic", "0"))}"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.VV_FW_RG.name}"
  depends_on          = ["azurerm_virtual_network.VV_FW_VNET",
                          "azurerm_public_ip.PublicIP_0"]

  ip_configuration {
    name                          = "${join("", list("ipconfig", "0"))}"
    subnet_id                     = "${azurerm_subnet.VV_FW_Subnet0.id}"
    private_ip_address_allocation = "dynamic"
    public_ip_address_id = "${azurerm_public_ip.PublicIP_0.id}"
  }

  tags {
    displayName = "${join("", list("NetworkInterfaces", "0"))}"
  }
}

resource "azurerm_network_interface" "VNIC1" {
  name                = "${join("", list("FW", "web-nic", "1"))}"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.VV_FW_RG.name}"
  depends_on          = ["azurerm_virtual_network.VV_FW_VNET"]

  ip_configuration {
    name                          = "${join("", list("ipconfig", "1"))}"
    subnet_id                     = "${azurerm_subnet.VV_FW_Subnet1.id}"
    private_ip_address_allocation = "dynamic"
  }

  tags {
    displayName = "${join("", list("NetworkInterfaces", "1"))}"
  }
}

resource "azurerm_network_interface" "VNIC2" {
  name                = "${join("", list("FW", "web-nic", "2"))}"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.VV_FW_RG.name}"
  depends_on          = ["azurerm_virtual_network.VV_FW_VNET"]

  ip_configuration {
    name                          = "${join("", list("ipconfig", "2"))}"
    subnet_id                     = "${azurerm_subnet.VV_FW_Subnet2.id}"
    private_ip_address_allocation = "dynamic"
  }

  tags {
    displayName = "${join("", list("NetworkInterfaces", "2"))}"
  }
}

resource "azurerm_virtual_machine" "VV_FW_Web" {
  name                  = "web-vm"
  location              = "${var.location}"
  resource_group_name   = "${azurerm_resource_group.VV_FW_RG.name}"
  vm_size               = "${var.gvmSize}"

  storage_image_reference {
    publisher = "${var.imagePublisher}"
    offer     = "${var.imageOffer}"
    sku       = "${var.ubuntuOSVersion}"
    version   = "latest"
  }

  storage_os_disk {
    name          = "web-osdisk"
    vhd_uri       = "${azurerm_storage_account.VV_FW_STG_AC.primary_blob_endpoint}vhds/osdisk-Web.vhd"
    caching       = "ReadWrite"
    create_option = "FromImage"
  }

  os_profile {
    computer_name  = "web-server-vm"
    admin_username = "${var.adminUsername}"
    admin_password = "${var.adminPassword}"
  }

  network_interface_ids = ["${azurerm_network_interface.VNIC0.id}",
                            "${azurerm_network_interface.VNIC1.id}",
                            "${azurerm_network_interface.VNIC2.id}"]
  primary_network_interface = "${azurerm_network_interface.VNIC0.id}"
  tags {
    environment = "staging"
  }
}

Debug Output

azurerm_virtual_machine.VV_FW_Web: Creating...
availability_set_id: "" => ""
delete_data_disks_on_termination: "" => "false"
delete_os_disk_on_termination: "" => "false"
license_type: "" => ""
location: "" => "westus"
name: "" => "web-vm"
network_interface_ids.#: "" => "3"
network_interface_ids.1300869970: "" => "/subscriptions/4b81abdf-236e-4012-a18d-4c5976220c92/resourceGroups/vvrg3/providers/Microsoft.Network/networkInterfaces/FWweb-nic0"
network_interface_ids.2743587454: "" => "/subscriptions/4b81abdf-236e-4012-a18d-4c5976220c92/resourceGroups/vvrg3/providers/Microsoft.Network/networkInterfaces/FWweb-nic2"
network_interface_ids.982418372: "" => "/subscriptions/4b81abdf-236e-4012-a18d-4c5976220c92/resourceGroups/vvrg3/providers/Microsoft.Network/networkInterfaces/FWweb-nic1"

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

The NIC's should be presented as: VNIC0, VNIC1, VNIC2.

Actual Behavior

The NIC's in the VM configure section were specified as VNIC0, VNIC1, VNIC2. However, when the NIC's are presented to the VM they are ordered as: VNIC0, VNIC2, VNIC1.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

[satishmohan]

Hi,
I'm experiencing the same issue - the interfaces are attached to the VM in an inconsistent order. Is there a workaround possible?

Thanks
Satish

[vinayvenkat]

Hi Satish, turns out there could be or there is a workaround. I'm not sure how deterministic it will be. But here is what I found.

I empirically over numerous trials identified that when three interfaces are specified, Terraform presents them as VNIC0, VNIC2, VNIC 1 for example. This was pretty consistent. So what I did was bit the bullet and then switched up the subnets that VNIC1 and VNIC2 were attached to. Hope this helps.

[satishmohan]

@vinayvenkat Thanks! Yes that seems to work - though I will have to do more trials. What does this mean though...? Is it deterministic but just the incorrect order - or is it indeterministic and we are just hitting on the correct hash lookup order....

[kblackstone]

It's random. Reversing the order in the code doesn't work reliably. This needs to be addressed. 1 launch may produce the correct results, a second launch may have 1 firewall with reversed interfaces and a third launch may have both firewalls with reversed interfaces. The order should be defined in the coding and be reliable, which right now it's not. Please fix this.

[satishmohan]

@kblackstone - on digging further it seems that the issue is with Azure itself. Azure doesn't seem to guarantee that the interfaces will be attached to the VM in deterministic order.

For instance, see this article: http://vaggeliskappas.com/2015/01/06/create-virtual-machine-multiple-nics-azure-iaas/

I tried fixing it in the AzureRM terraform provider to make the order deterministic - but it didn't help. They still show up in random order.

I ended up fixing it in the boot scripts inside my VM - figure out which subnet a particular interface is attached to, and use that in the scripts.

HTH,
Satish

[kblackstone]

Appreciate you digging into this. Any idea why an ARM template would provide more consistent results?

[tombuildsstuff]

Fixed in #426

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!