Error destroying VM - Operation 'startTenantUpdate' is not allowed on VM #365

dbilleci-lightstream · 2019-09-13T02:05:25Z

Hello, I felt that this issue was probably an Azure API issue, but a nice Microsoft folk over there told me that I should look at Pulumi and or Terraform.

Azure/azure-rest-api-specs#7207

Boils down to the fact that code that has been working for some time suddenly is erroring on teardown of simple resources. Another user confirmed similar behavior, both of them dealing with attempting to destroy a NIC association that throws an error.

Simple deployed resources linking to an existing LB:

azure:core:ResourceGroup
azure:network:NetworkInterface
azure:network:NetworkInterfaceBackendAddressPoolAssociation
azure:compute:VirtualMachine

Then try to destroy:

  pulumi:pulumi:Stack (compute-api-compute-api-b-stage-eus2):
    error: update failed

  azure:network:NetworkInterfaceBackendAddressPoolAssociation (api-B-P-0-Stage-Eus2-Assoc):
    error: Plan apply failed: deleting urn:pulumi:compute-api-b-stage-eus2::compute-api::azure:network/networkInterfaceBackendAddressPoolAssociation:NetworkInterfaceBackendAddressPoolAssociation::api-B-P-0-Stage-Eus2-Assoc: Error waiting for removal of Backend Address Pool Association for NIC "api-B-Vm-0-Stage-Eus2-Nic" (Resource Group "api-B-Stage-Eus2-Rg"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'api-B-0-Stage-Eus2-Vm' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]

Quote from Microsoft guy:

I think this should be raised on terraform or pulumi SDK.

...

This API spec is for everything from Azure portal to CLI/powershell/go/java/dotnet client. If this got problem everyone is in trouble hence it is unlikely the cause of your issue (personally I just deleted a few resources on portal and see no issue).

I got this error behavior on older Azure provider (pre 1.0.0) and then upgraded to 1.0.0 today to see if it fixed the issue, it did not for new deploys also.. still happening.

If there is a better place to submit this, please let me know.

VERSIONS:

deploy@240aa74ed033:/data$ pulumi version
v1.1.0
deploy@240aa74ed033:/data$ pulumi plugin ls
NAME   KIND      VERSION  SIZE    INSTALLED  LAST USED
aws    resource  1.2.0    218 MB  n/a        6 hours ago
azure  resource  1.0.0    155 MB  n/a        6 hours ago

The text was updated successfully, but these errors were encountered:

mikhailshilkov · 2019-09-13T02:47:01Z

Hi @dbilleci-lightstream thank you for reporting this.
Is there a snippet of the program to reproduce the problem, that you could share?

dbilleci-lightstream · 2019-09-13T03:13:38Z

Ah, sure.. it will take me a bit, it's in the middle of our automation and I'll have to recreate it, but I don't think it will be too hard.

Other users on other repos are reporting similar issues also: openshift/installer#2353

dbilleci-lightstream · 2019-09-13T18:48:27Z

OK - I've got sample code now:

import * as azure from "@pulumi/azure";

var resource_group_container_name = 'test-bug'
var resource_group_container = new azure.core.ResourceGroup(resource_group_container_name, {
    location: 'EastUS2',
    name: resource_group_container_name
});

var nic_name = 'test-nic';
var vm_nic = new azure.network.NetworkInterface(nic_name, {
    name: nic_name,
    location: 'EastUS2',
    resourceGroupName: resource_group_container_name,
    enableIpForwarding: false,
    enableAcceleratedNetworking: false,
    ipConfigurations: [{
        name: 'ipconfig',
        subnetId: '/subscriptions/11111111-22222-33333-444444/resourceGroups/VirtualNetwork-Stage-Eus2-Rg/providers/Microsoft.Network/virtualNetworks/MyCompany-Stage-Eus2-Vnet/subnets/Web-Stage-Eus2-Subnet',
        privateIpAddressAllocation: 'dynamic'
    }],
    networkSecurityGroupId: '/subscriptions/11111111-22222-33333-444444/resourceGroups/NetworkSecurityGroup-Stage-Eus2-Rg/providers/Microsoft.Network/networkSecurityGroups/API-Stage-Eus2-Nsg'
},
    {
        dependsOn: resource_group_container
    }
);

var nic_backend_association_name = 'test-nic-assoc';
var nic_backend_association = new azure.network.NetworkInterfaceBackendAddressPoolAssociation(nic_backend_association_name, {
    networkInterfaceId: vm_nic.id,
    ipConfigurationName: 'ipconfig',
    backendAddressPoolId: '/subscriptions/11111111-22222-33333-444444/resourceGroups/PublicLoadBalancer-Stage-Eus2-Rg/providers/Microsoft.Network/loadBalancers/API-Stage-Eus2-Plb/backendAddressPools/API-static-be'
},
{
    dependsOn: [
        vm_nic,
        resource_group_container
    ]
}
);

var vm_name = 'testy-test-vm-bug';
var vm = new azure.compute.VirtualMachine(vm_name, {
    name: vm_name,
    location: 'EastUS2',
    resourceGroupName: resource_group_container_name,
    vmSize: 'Standard_DS1_v2',
    networkInterfaceIds:[vm_nic.id],
    deleteOsDiskOnTermination: true,
    osProfile: {
        computerName: vm_name.toLowerCase(),
        adminUsername: 'ubuntu',
    },
    osProfileLinuxConfig: {
        disablePasswordAuthentication: true,
        sshKeys: [{
            path: '/home/ubuntu/.ssh/authorized_keys',
            keyData: "ssh-rsa AAAAB3NzaC1ycXXXXXXXXXXXXX"
        }]
    },
    storageOsDisk: {
        name: `${vm_name}-0-Disk`,
        osType: 'Linux',
        caching: 'ReadWrite',
        diskSizeGb: 30,
        createOption: 'FromImage',
        managedDiskType: 'Standard_LRS'
    },
    storageImageReference: {
        id: '/subscriptions/11111111-22222-33333-444444/resourceGroups/SharedImageGallery-Prod-Eus2-Rg/providers/Microsoft.Compute/galleries/MyCompanyProdEus2Gallery/images/API/versions/19082617.55.31'
    },
    bootDiagnostics: {
        enabled: true,
        storageUri: `https://mycompanydiageus2.blob.core.windows.net/`
    }
}, {
    dependsOn: [
        resource_group_container
    ]
});

When executed, produces:

deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi up
Previewing update (dev):

     Type                                                            Name               Plan       
 +   pulumi:pulumi:Stack                                             test-bug-dev       create     
 +   ├─ azure:core:ResourceGroup                                     test-bug           create     
 +   ├─ azure:network:NetworkInterface                               test-nic           create     
 +   ├─ azure:compute:VirtualMachine                                 testy-test-vm-bug  create     
 +   └─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc     create     
 
Resources:
    + 5 to create

Do you want to perform this update? yes
Updating (dev):

     Type                                                            Name               Status      
 +   pulumi:pulumi:Stack                                             test-bug-dev       created     
 +   ├─ azure:core:ResourceGroup                                     test-bug           created     
 +   ├─ azure:network:NetworkInterface                               test-nic           created     
 +   ├─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc     created     
 +   └─ azure:compute:VirtualMachine                                 testy-test-vm-bug  created     
 
Resources:
    + 5 created

Duration: 1m49s

Permalink: https://app.pulumi.com/dbilleci-lightstream/test-bug/dev/updates/7
deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi destroy
Previewing destroy (dev):

     Type                                                            Name               Plan       
 -   pulumi:pulumi:Stack                                             test-bug-dev       delete     
 -   ├─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc     delete     
 -   ├─ azure:compute:VirtualMachine                                 testy-test-vm-bug  delete     
 -   ├─ azure:network:NetworkInterface                               test-nic           delete     
 -   └─ azure:core:ResourceGroup                                     test-bug           delete     
 
Resources:
    - 5 to delete

Do you want to perform this destroy? yes
Destroying (dev):

     Type                                                            Name               Status                  Info
     pulumi:pulumi:Stack                                             test-bug-dev       **failed**              1 error
 -   ├─ azure:compute:VirtualMachine                                 testy-test-vm-bug  deleted                 
 -   └─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc     **deleting failed**     1 error
 
Diagnostics:
  azure:network:NetworkInterfaceBackendAddressPoolAssociation (test-nic-assoc):
    error: Plan apply failed: deleting urn:pulumi:dev::test-bug::azure:network/networkInterfaceBackendAddressPoolAssociation:NetworkInterfaceBackendAddressPoolAssociation::test-nic-assoc: Error waiting for removal of Backend Address Pool Association for NIC "test-nic" (Resource Group "test-bug"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'testy-test-vm-bug' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]
 
  pulumi:pulumi:Stack (test-bug-dev):
    error: update failed
 
Resources:
    - 1 deleted

Duration: 3m37s

Permalink: https://app.pulumi.com/dbilleci-lightstream/test-bug/dev/updates/8
deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi version
v1.1.0
deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi plugin ls
NAME   KIND      VERSION  SIZE    INSTALLED  LAST USED
aws    resource  1.2.0    218 MB  n/a        23 hours ago
azure  resource  1.0.0    155 MB  n/a        23 hours ago

TOTAL plugin cache size: 373 MB

mikhailshilkov · 2019-09-13T19:18:33Z

@dbilleci-lightstream Thank you so much for the great repro.

So it looks like the VM is deleted first despite its dependency on NIC and the association. Will it help if you add vm_nic and nic_backend_association to the dependsOn list of the VM before you create those resources? This shouldn't be required, but it looks like the implicit dependency isn't tracked properly.

dbilleci-lightstream · 2019-09-13T21:46:17Z

OK, I'll try this in a just a moment. But, wanted to bring up what seems like another related issue, which may or may not be solved by this dependency adjustment.

When it fails in this fashion as shown above, the easiest way to start over is to simply destroy the resource group.

After destroying the resource group, if you try to pulumi refresh in order to clear the items out of the state file, it fails:

deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi refresh
Previewing refresh (dev):

     Type                                                            Name            Plan        Info
     pulumi:pulumi:Stack                                             test-bug-dev                1 error
 -   ├─ azure:core:ResourceGroup                                     test-bug        delete
 ~   ├─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc  refresh     1 error
 -   └─ azure:network:NetworkInterface                               test-nic        delete

Diagnostics:
  pulumi:pulumi:Stack (test-bug-dev):
    error: preview failed

  azure:network:NetworkInterfaceBackendAddressPoolAssociation (test-nic-assoc):
    error: Preview failed: refreshing urn:pulumi:dev::test-bug::azure:network/networkInterfaceBackendAddressPoolAssociation:NetworkInterfaceBackendAddressPoolAssociation::test-nic-assoc: Network Interface "test-nic" (Resource Group "test-bug") was not found!

The azure:network:NetworkInterfaceBackendAddressPoolAssociation has the RG listed as a dependency, and the RG is shown as deleted by Pulumi, so the fact that it then fails on trying to refresh the azure:network:NetworkInterfaceBackendAddressPoolAssociation is odd, isn't it?

dbilleci-lightstream · 2019-09-13T22:14:47Z

OK so adding the extra 2 dependencies as suggested allowed me to update and destroy - so thanks for that guidance.

However, I then deployed the test infra one more time and tried the manual RG delete again and refresh, and that still failed:

deploy@3f097c3b9d2c:/data/pulumi/infra/azure/test-bug$ pulumi refresh
Previewing refresh (dev):

     Type                                                            Name            Plan        Info
     pulumi:pulumi:Stack                                             test-bug-dev                1 error
 -   ├─ azure:core:ResourceGroup                                     test-bug        delete      
 ~   ├─ azure:network:NetworkInterfaceBackendAddressPoolAssociation  test-nic-assoc  refresh     1 error
 -   └─ azure:network:NetworkInterface                               test-nic        delete      
 
Diagnostics:
  pulumi:pulumi:Stack (test-bug-dev):
    error: preview failed
 
  azure:network:NetworkInterfaceBackendAddressPoolAssociation (test-nic-assoc):
    error: Preview failed: refreshing urn:pulumi:dev::test-bug::azure:network/networkInterfaceBackendAddressPoolAssociation:NetworkInterfaceBackendAddressPoolAssociation::test-nic-assoc: Network Interface "test-nic" (Resource Group "test-bug") was not found!

Seems like there is some odd dependency stuff happening.

For clarity, where is the core issue here (terraform provider, azure CLI change, pulumi internals)? And any idea why it suddenly cropped up?

Thanks!

mikhailshilkov · 2019-09-16T07:57:18Z

@dbilleci-lightstream I ported your example to terraform and was able to reproduce both issues there.

The refresh of deleted resource issue is hashicorp/terraform-provider-azurerm#2491
I created a new issue for original issue: hashicorp/terraform-provider-azurerm#4330

If you agree those are the same issues, please upvote them and let's track the progress there.

dbilleci-lightstream · 2019-09-16T15:31:40Z

Thank you @mikhailshilkov - I've upvoted both issues.

mikhailshilkov · 2019-09-16T15:40:51Z

You are welcome @dbilleci-lightstream
With a workaround available (set the dependency explicitly), I'll close this issue for now and let's track those two items upstream.

dbilleci-lightstream mentioned this issue Sep 13, 2019

Change In API behavior? Azure/azure-rest-api-specs#7207

Closed

mikhailshilkov closed this as completed Sep 16, 2019

mikhailshilkov mentioned this issue Jan 17, 2020

Does not Delete NIC when replacing a VirtualMachine #441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error destroying VM - Operation 'startTenantUpdate' is not allowed on VM #365

Error destroying VM - Operation 'startTenantUpdate' is not allowed on VM #365

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

mikhailshilkov commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019

mikhailshilkov commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

mikhailshilkov commented Sep 16, 2019

dbilleci-lightstream commented Sep 16, 2019

mikhailshilkov commented Sep 16, 2019 •

edited

Loading

Error destroying VM - Operation 'startTenantUpdate' is not allowed on VM #365

Error destroying VM - Operation 'startTenantUpdate' is not allowed on VM #365

Comments

dbilleci-lightstream commented Sep 13, 2019 • edited Loading

mikhailshilkov commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019

mikhailshilkov commented Sep 13, 2019

dbilleci-lightstream commented Sep 13, 2019 • edited Loading

dbilleci-lightstream commented Sep 13, 2019 • edited Loading

mikhailshilkov commented Sep 16, 2019

dbilleci-lightstream commented Sep 16, 2019

mikhailshilkov commented Sep 16, 2019 • edited Loading

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

dbilleci-lightstream commented Sep 13, 2019 •

edited

Loading

mikhailshilkov commented Sep 16, 2019 •

edited

Loading