Struggle to update a cosmos db account

[mikhailshilkov]

This looks like several unrelated issues, but I decided to bump them all here.

I had a Pulumi program which defined a Cosmos DB account with 1 region:

    location: "West Europe",
    geoLocations: [
        {
            location: "West Europe",
            failoverPriority: 0
        }
      ]

pulumi up worked and everything was ok.

Now, I changed the account to have two replicas:

    location: "West Europe",
    geoLocations: [
        {
            location: "West Europe",
            failoverPriority: 0
        },
        {
            location: "North Europe",
            failoverPriority: 1
        },
    ],

Problem 1. pulumi up now suggests replacing my existing account with data with a new one:

+-  ├─ azure:cosmosdb:Account        mygeocosmos  replace     [diff: ~geoLocations,name]

the reason is that the name changed. Why is that?

Okay, I proceed with the change. Creating an account takes time.

Problem 2. After ~10 minutes pulumi quits with this message:

Sounds sort of similar to #217 but with a very different error message.

Problem 3. In fact, my Cosmos DB account is created. But when I run pulumi up it renames the account again and starts creating a new one (a third one).

Eventually, I succeeded to create an updated Cosmos account. However, pulumi keeps suggesting to replace it on every up command.

A workaround of specifying the full name explicitly seems to help.

[lukehoban]

Problem 2 looks like another manifestation of hashicorp/terraform-provider-azurerm#2602.

[lukehoban]

I looked into (1) a bit.

The addition of the second element of geoLocations does indeed lead Pulumi to try to do a replacement. With verbose logs, we see that this was in fact due to a percieved change in the geoLocations property. The logs we get back include:

I0501 13:13:08.412888   99719 rpc.go:68] Marshaling property for RPC[Provider[azure, 0xc000f68550].Diff(urn:pulumi:dev::cosmosdb-219::azure:cosmosdb/account:Account::cosmosDb,/subscriptions/8bafcca2-660a-4459-a503-b785cf317a3a/resourceGroups/resourcegroup432f78f7/providers/Microsoft.DocumentDB/databaseAccounts/cosmosdb63f86d3a).olds]: geoLocations={[{map[failoverPriority:{0} id:{cosmosdb63f86d3a-westus} location:{westus} prefix:{}]}]}
I0501 13:13:08.413306   99719 rpc.go:68] Marshaling property for RPC[Provider[azure, 0xc000f68550].Diff(urn:pulumi:dev::cosmosdb-219::azure:cosmosdb/account:Account::cosmosDb,/subscriptions/8bafcca2-660a-4459-a503-b785cf317a3a/resourceGroups/resourcegroup432f78f7/providers/Microsoft.DocumentDB/databaseAccounts/cosmosdb63f86d3a).news]: geoLocations={[{map[__defaults:{[]} failoverPriority:{0} location:{WestUS}]} {map[__defaults:{[]} failoverPriority:{1} location:{EastUS}]}]}
I0501 13:13:08.416693   99719 provider_plugin.go:308] Provider[azure, 0xc000f68550].Diff(urn:pulumi:dev::cosmosdb-219::azure:cosmosdb/account:Account::cosmosDb,/subscriptions/8bafcca2-660a-4459-a503-b785cf317a3a/resourceGroups/resourcegroup432f78f7/providers/Microsoft.DocumentDB/databaseAccounts/cosmosdb63f86d3a) success: changes=2 #replaces=[geoLocations] #stables=[name location resourceGroupName kind] delbefrepl=false, diffs=#[geoLocations geoLocations readEndpoints enableAutomaticFailover name secondaryMasterKey connectionStrings geoLocations geoLocations geoLocations secondaryReadonlyMasterKey location resourceGroupName writeEndpoints consistencyPolicy kind geoLocations tags consistencyPolicy primaryReadonlyMasterKey endpoint offerType geoLocations primaryMasterKey enableMultipleWriteLocations consistencyPolicy geoLocations geoLocations isVirtualNetworkFilterEnabled consistencyPolicy]
I0501 13:13:08.418051   99719 step_generator.go:314] Planner decided to replace 'urn:pulumi:dev::cosmosdb-219::azure:cosmosdb/account:Account::cosmosDb' (oldprops=map[__defaults:{[{enableMultipleWriteLocations} {isVirtualNetworkFilterEnabled} {name}]} consistencyPolicy:{map[__defaults:{[]} consistencyLevel:{BoundedStaleness} maxIntervalInSeconds:{10} maxStalenessPrefix:{200}]} enableAutomaticFailover:{true} enableMultipleWriteLocations:{false} geoLocations:{[{map[__defaults:{[]} failoverPriority:{0} location:{WestUS}]}]} isVirtualNetworkFilterEnabled:{false} kind:{MongoDB} location:{westus} name:{cosmosdb63f86d3a} offerType:{Standard} resourceGroupName:{resourcegroup432f78f7}] inputs=map[__defaults:{[{enableMultipleWriteLocations} {isVirtualNetworkFilterEnabled} {name}]} consistencyPolicy:{map[__defaults:{[]} consistencyLevel:{BoundedStaleness} maxIntervalInSeconds:{10} maxStalenessPrefix:{200}]} enableAutomaticFailover:{true} enableMultipleWriteLocations:{false} geoLocations:{[{map[__defaults:{[]} failoverPriority:{0} location:{WestUS}]} {map[__defaults:{[]} failoverPriority:{1} location:{EastUS}]}]} isVirtualNetworkFilterEnabled:{false} kind:{MongoDB} location:{westus} name:{cosmosdb7e64da1b} offerType:{Standard} resourceGroupName:{resourcegroup432f78f7}])

Note in particular replaces=[geoLocations].

That said, looking at the Terraform Schema for this, I cannot explain why this change would trigger a replacement. I'll need to debug a little deeper. There are a few slightly unusual things happening around normalization (both diff suppression and statefile storage normalization) for the location property - which could be contributing here.

[lukehoban]

For reference - here is the repro:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

// Create an Azure Resource Group
const resourceGroup = new azure.core.ResourceGroup("resourceGroup", {
    location: "WestUS",
});

// Create an Azure resource (Storage Account)
const account = new azure.storage.Account("storage", {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
    accountTier: "Standard",
    accountReplicationType: "LRS",
});


const cosmosdb = new azure.cosmosdb.Account("cosmosDb", {
    kind: "MongoDB",
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
    consistencyPolicy: {
        consistencyLevel: "BoundedStaleness",
        maxIntervalInSeconds: 10,
        maxStalenessPrefix: 200
    },
    offerType: "Standard",
    enableAutomaticFailover: true,
    geoLocations: [
        { location: "WestUS", failoverPriority: 0 },
        // { location: "EastUS", failoverPriority: 1 }
    ]
});

// Export the connection string for the storage account
export const endpoint = cosmosdb.endpoint;

[lukehoban]

The issue in step (1) is a bug in the upstream Terraform AzureRM provider. The location field on geo_location element types is set as ForceNew, which causes any addition of a geo location to trigger a replacement (Terraform's internal diff treats "not present" => "EastUS" as a diff which triggers the ForceNew - very unfortunate since that is almost never what this transition has to imply at the cloud provider level). The same change will trigger a replacement in Terraform/HCL.

The issue in step (2) is hashicorp/terraform-provider-azurerm#2602.

The issue in step (3) is expected - though not ideal - if the provider does not return back a resource id, there is no way for Pulumi to know what was created. So due to the bug in (2), we never get told what resource was created, and it gets orphaned. There is nothing Pulumi can really do here - we just need to avoid having bugs like (2) in the providers.

[mikhailshilkov]

Thanks for your analysis @lukehoban
Is there an open upstream issue for (1)?
They also seem to add regions one-by-one, which adds to the total time it takes to provision the full cluster and contributes to (2).

[lukehoban]

Is there an open upstream issue for (1)?

Opened hashicorp/terraform-provider-azurerm#3532.

They also seem to add regions one-by-one, which adds to the total time it takes to provision the full cluster and contributes to (2).

Definitely feel free to open an issue and/or PR on that upstream as well!