stopped virtual machine in terraform

[hashibot]

This issue was originally opened by @retheshnair as hashicorp/terraform#16629. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Currently when I stop VM terraform doesn't change the state in azure

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please try upgrading because your issue may have already been fixed.

[root@puppettest PUPPETPREPROD]# terraform -v
Terraform v0.10.8

[root@puppettest PUPPETPREPROD]#

[tombuildsstuff]

hey @retheshnair

Thanks for opening this issue.

The AzureRM Terraform Provider doesn't currently support managing the State of Virtual Machines (e.g. Running/Stopped) - so that we can understand the issue better, would it be possible to get some more information about the issue you're facing?

Thanks!

[clippermadness]

@tombuildsstuff I may be seeing the same or a similar issue with stopped (deallocated) VMs in Azure and terraform. I have three machines that I keep stopped and launch programatically based on data availability in my application. If they are stopped, terraform plan reports:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ azurerm_virtual_machine.pipeline_audprocessor_dev2
      storage_os_disk.0.disk_size_gb:      "0" => "32"
      storage_os_disk.0.managed_disk_type: "" => "Standard_LRS"

  ~ azurerm_virtual_machine.pipeline_vidprocessor_faces_dev2
      storage_os_disk.0.disk_size_gb:      "0" => "32"
      storage_os_disk.0.managed_disk_type: "" => "Standard_LRS"

  ~ azurerm_virtual_machine.pipeline_vidprocessor_frames_dev2
      storage_os_disk.0.disk_size_gb:      "0" => "32"
      storage_os_disk.0.managed_disk_type: "" => "Standard_LRS"


Plan: 0 to add, 3 to change, 0 to destroy.

When I run terraform apply (just for giggles), I get errors:

Error: Error applying plan:

3 error(s) occurred:

* azurerm_virtual_machine.pipeline_audprocessor_dev2: 1 error(s) occurred:

* azurerm_virtual_machine.pipeline_audprocessor_dev2: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Managed disk storage account type change through Virtual Machine 'aud1-dev2' is not allowed. Please update disk resource at /subscriptions/5e384bd8-c8ba-4040-b29d-4e37a8921981/resourceGroups/development/providers/Microsoft.Compute/disks/aud1-dev2-osdisk."
* azurerm_virtual_machine.pipeline_vidprocessor_frames_dev2: 1 error(s) occurred:

* azurerm_virtual_machine.pipeline_vidprocessor_frames_dev2: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Managed disk storage account type change through Virtual Machine 'frames1-dev2' is not allowed. Please update disk resource at /subscriptions/5e384bd8-c8ba-4040-b29d-4e37a8921981/resourceGroups/development/providers/Microsoft.Compute/disks/frames1-dev2-osdisk."
* azurerm_virtual_machine.pipeline_vidprocessor_faces_dev2: 1 error(s) occurred:

* azurerm_virtual_machine.pipeline_vidprocessor_faces_dev2: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Managed disk storage account type change through Virtual Machine 'faces1-dev2' is not allowed. Please update disk resource at /subscriptions/5e384bd8-c8ba-4040-b29d-4e37a8921981/resourceGroups/development/providers/Microsoft.Compute/disks/faces1-dev2-osdisk."

It seems like the provider should understand the machine state enough to not plan changes when managed machines are stopped.

Terraform v0.10.8

• provider.aws: version = "~> 1.2"
• provider.azurerm: version = "~> 0.3"

[retheshnair]

@tombuildsstuff yep @clippermadness has put the details . Still i will try to put more details

Case -1

After provisioning the machine using terraform I have stop it from portal or azcli since terrafrom don't have options of specifying the state . Something like state = deallocated or stopped . Once the VM is deallocated if we try to do terraform plan it is showing a in-place update since the disk is Reserved but not leased

Case-2
Suppose we have stopped the VM from OS level using init 0 in linux or shutdown command from windows . This VM be won't dellocated hence it will incur charges but the state is stopped . When do a terraform plan show the infrastructure is up to date but the VM is still in stopped state

We should have something like state = stopped or deallocated in azurerm_virtual_machine to manage the state from terraform

[root@puppettest RETHESH_PUPPETPREPROD]# terraform  -v
Terraform v0.11.0
+ provider.azurerm v0.3.2
```

[root@puppettest RETHESH_PUPPETPREPROD]#

Terraform Plan ( Deallocated )

---

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

~ azurerm_virtual_machine.puppetserver_vms
storage_os_disk.0.disk_size_gb: "0" => "50"
storage_os_disk.0.managed_disk_type: "" => "Standard_LRS"

Plan: 0 to add, 1 to change, 0 to destroy.

---

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Terraform Apply ( Dealloacted)

storage_os_disk.0.disk_size_gb: "0" => "50"
storage_os_disk.0.managed_disk_type: "" => "Standard_LRS"

Error: Error applying plan:

1 error(s) occurred:

• azurerm_virtual_machine.puppetserver_vms: 1 error(s) occurred:

• azurerm_virtual_machine.puppetserver_vms: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Managed disk storage account type change through Virtual Machine 'testimage01' is not allowed. Please update disk resource at /subscriptions/4bd70f6c-5b53-4a38-9211-993e21b6f5e0/resourceGroups/testDevOpsPreProd/providers/Microsoft.Compute/disks/testimage01_osdisk01."

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[root@puppettest RETHESH_PUPPETPREPROD]#

Terraform Plan (still incurring compute charges)

---

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

[retheshnair]

@tombuildsstuff . with the ARM API you can't get the VM state in the List ALL VMs API. You need to call the instance view API on each VM to get this.

https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Compute/virtualMachines/{vm-name}/InstanceView?api-version={api-version}

This API call will fetch the

https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/testDevOpsPreProd/providers/Microsoft.Compute/virtualMachines/testimage01/InstanceView?api-version=2017-03-30

VM running
===========

"statuses": [
    {
      "code": "ProvisioningState/succeeded",
      "level": "Info",
      "displayStatus": "Provisioning succeeded",
      "time": "2017-11-18T20:18:21.0975509+00:00"
    },
    {
      "code": "PowerState/running",
      "level": "Info",
      "displayStatus": "VM running"
    }
  ]


VM dellocated 
==========

"statuses": [
    {
      "code": "ProvisioningState/succeeded",
      "level": "Info",
      "displayStatus": "Provisioning succeeded",
      "time": "2017-11-18T20:22:53.2373216+00:00"
    },
    {
      "code": "PowerState/deallocated",
      "level": "Info",
      "displayStatus": "VM deallocated"
    }
  ]

VM Status Getting update
=====================

"statuses": [
    {
      "code": "ProvisioningState/updating",
      "level": "Info",
      "displayStatus": "Updating"
    },
    {
      "code": "PowerState/starting",
      "level": "Info",
      "displayStatus": "VM starting"
    }
  ]

VM Status ( Stopping ) 
=============

"statuses": [
    {
      "code": "ProvisioningState/succeeded",
      "level": "Info",
      "displayStatus": "Provisioning succeeded",
      "time": "2017-11-18T20:29:24.0967114+00:00"
    },
    {
      "code": "PowerState/stopping",
      "level": "Info",
      "displayStatus": "VM stopping"
    }
  ]

VM STOPPED
============
"statuses": [
    {
      "code": "ProvisioningState/succeeded",
      "level": "Info",
      "displayStatus": "Provisioning succeeded",
      "time": "2017-11-18T20:29:24.0967114+00:00"
    },
    {
      "code": "PowerState/stopped",
      "level": "Info",
      "displayStatus": "VM stopped"
    }
  ]

On AWS we can get the description of instances and get the status, but on Azure we need to send an individual request for every instance just to find out if they are running, deallocated or stopped.

Please refer this link for more details .

[metacpp]

This closes #539 due to duplication.

[JunyiYi]

Hi @retheshnair , from the Azure Virtual Machine service team:

A deallocated (stopped) virtual machine will not return any storage information.

You can also confirm the behavior mentioned above in Azure portal. So I think you should not run any terraform commands when the virtual machine is turned off.

[JunyiYi]

Hi @retheshnair and @clippermadness , just to let you know the code fix has been merged, and it will be published in 1.4.0. I will close the issue for now, if you meet any problems in the future, feel free to reopen it or create a new issue. Thanks.

[tombuildsstuff]

hey @retheshnair @clippermadness

Just to let you know that v1.4.0 of the Azure Provider has been released which includes the fix for this :)

Thanks!

[rem-aj]

Good job @tombuildsstuff , great timing for my team! Now we can actually shut down servers and not worry about terraform thinking they don't exist at all!

[clippermadness]

@tombuildsstuff Confirmed: works. Thanks!

[JohnDelisle]

@tombuildsstuff I just tried to update managed OS disks on VMs that are powered down (stopped/ deallocated via portal) and they still fail. I get the same error even when they're up and running, actually.

plan:
~ managed_disk_type = "Standard_LRS" -> "StandardSSD_LRS"

error:

Error: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status= Code="OperationNotAllowed" Message="Managed disk storage account type change through Virtual Machine 'infrajumpp02' is not allowed. Please update disk resource at /subscriptions/123123/resourceGroups/rg/providers/Microsoft.Compute/disks/infrajumpp02-os." Target="osDisk.managedDisk.storageAccountType"

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

[tombuildsstuff]

@JohnDelisle the azurerm_virtual_machine resource is being superseded in version 2.0 by the azurerm_linux_virtual_machine and azurerm_windows_virtual_machine resources which support these delta updates - these are available as an opt-in Beta in 1.x versions of the Azure Provider but will be going GA in 2.0.

Both of these new resources support (amongst other things) resizing the OS Disk, changing the Storage Type - which will stop/start the virtual machine as needed to make these changes (and should work with the VM in a stopped state). Unfortunately we have no plans to backport this to the existing azurerm_virtual_machine resource (for reasons outlined in #2807) - however I believe the azurerm_linux_virtual_machine and azurerm_windows_virtual_machine resources should provide a solution to this going forwards

Thanks!