azurerm_postgresql_database.testdb: Code="InternalServerError" while destroying

[svangeti]

Terraform (and AzureRM Provider) Version

Terraform -v = 0.11.8
azurerm: version = "~> 1.30"

Affected Resource(s)

• azurerm_postgresql_database

Terraform Configuration Files

{
  "resource" : {
    "azurerm_postgresql_server" : {
      "pgserver" : {
        "ssl_enforcement" : "Enabled",
        "administrator_login" : "${var.serveruser}",
        "administrator_login_password" : "${var.serverpassword}",
        "name" : "${var.pgservers[2]}",
        "storage_profile" : {
          "storage_mb" : "${var.storage_mb}",
          "backup_retention_days" : "${var.backup_retention_days}",
          "geo_redundant_backup" : "${var.geo_redundant_backup}"
        },
        "resource_group_name" : "${var.rgName}",
        "location" : "${var.location}",
        "sku" : {
          "family" : "${var.skufamily}",
          "name" : "${var.skuname}",
          "capacity" : "${var.skucapacity}",
          "tier" : "${var.skutier}"
        },
        "version" : "${var.pgversion}"
      }
    }
  }
}

{
  "resource" : {
    "azurerm_postgresql_database" : {
      "testdb" : {
        "charset" : "UTF8",
        "server_name" : "${azurerm_postgresql_server.pgserver.name}",
        "name" : "${var.pgDBs[2]}",
        "resource_group_name" : "${var.resource_group_name}",
        "collation" : "en_US"
      }
    }
  }
}

Debug Output

Expected Behavior

It should destroy the resources from Azure

Actual Behavior

Error: Error applying plan:
1 error(s) occurred:

• azurerm_postgresql_database.testdb (destroy): 1 error(s) occurred:
• azurerm_postgresql_database.testdb: Code="InternalServerError" Message="An unexpected error occured while processing the request. Tracking ID: '5d3cfe7f-d576-4046-a48e-3f587064ed13'"
  Terraform does not automatically rollback in the face of errors.

Steps to Reproduce

1. terraform apply

2. terraform destroy

[tombuildsstuff]

hi @svangeti

Thanks for opening this issue.

Taking a look into this unfortunately this is a temporary error being returned by Azure - the Azure Provider automatically retries certain status codes (such as Internal Server Error) to work around bugs in the API; which means that for this to be raised this status code must have been returned repeatedly.

Where we receive Internal Server Errors there's not much we can do but repeatedly retry the request (as we're doing) and then eventually give up (which we're doing after several attempts) - as such in this case I don't believe there's much more we can do to work around this unfortunately.

Since this was a temporary error from Azure I believe this should now be resolved and work as normal (our Acceptance Tests are passing for Postgres too) - would you be able to take a look and see if this now works for you? As I believe this should now be resolved I'm going to close this issue for the moment but please let us know if that still doesn't work for you and we'll take another look :)

Thanks!

[wasfree]

Hi @tombuildsstuff

we can reproduce this issue for a while in westeurope region. According to some tests this seems only be the case if we create a azurerm_postgresql_server in combination with azurerm_sql_database (but not 100% sure if this is releated). Agree with you this seems to be related to Azure, so we will open a Support ticket there.

TestPRV_Postgres 2019-10-10T11:46:02Z command.go:158: module.postgres.azurerm_postgresql_database.pg_db["unit-tests-tfm-azure-postgres-db-1"]: Creating...
TestPRV_Postgres 2019-10-10T11:46:12Z command.go:158: module.postgres.azurerm_postgresql_database.pg_db["unit-tests-tfm-azure-postgres-db-1"]: Still creating... [10s elapsed]
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158: 
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158: Error: Code="InternalServerError" Message="An unexpected error occured while processing the request. Tracking ID: '36de0215-189f-482e-9be7-c03588812ea4'"
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158: 
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158:   on ../main.tf line 27, in resource "azurerm_postgresql_database" "pg_db":
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158:   27: resource "azurerm_postgresql_database" "pg_db" {
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158: 
TestPRV_Postgres 2019-10-10T11:46:18Z command.go:158: 

[svangeti]

@tombuildsstuff / @NickMetz , In my case the issue was the database session being open and not disconnected. If you disconnect all DB sessions, the error would go away.

[wasfree]

@svangeti this issue is more related to database collation. This is what happens on Azure backend side:

The request failed :

'Microsoft.RDMS.InstanceAgent.Service.Exceptions.InstanceAgentException: The database already exists and the collation cannot be changed. Expect: de-DE, Current: de_DE. ---> Npgsql.PostgresException: 42P04: database "devdb" already exists
   at Npgsql.NpgsqlConnector.<DoReadMessageAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlConnector.<ReadMessageWithPrependedAsync>d__5.MoveNext()

Unfortunately it seems that Azure uses collations with hyphen instead of underscore. This was also topic of an old issue #396. Looking into terraform documentation shows that we sent db collations to API with underscore. Azure has a wrapper and will transform e.g. de_DE to de-DE, so we should not see any issue at this point. But my guess is this process is failing in case of retries and other unknows triggers on Azure side.

User-Agent: Go/go1.12.6 (amd64-linux) go-autorest/v13.0.0 Azure-SDK-For-Go/v33.2.0 postgresql/2017-12-01 Ha
shiCorp Terraform/0.12.9 (+https://www.terraform.io) terraform-provider-azurerm/1.35.0
Authorization: **REDACTED**
{"properties":{"charset":"UTF8","collation":"de_DE"}}

So from my prospective we have two issues:

1. terraform is only accepting collation with underscore (validation) and we overwrite the response from API in DatabaseRead.

if props := resp.DatabaseProperties; props != nil {
		d.Set("charset", props.Charset)

		if collation := props.Collation; collation != nil {
			v := strings.Replace(*collation, "-", "_", -1)
			d.Set("collation", v)
		}
	}

2. Azure process to transform underscore to hyphen has issues sometimes

Solution:

• Terraform should sent collation with hyphen instead underscore.

• This will prevent us from this transformation issue. I don't see the reason why terraform is enforcing underscore for collation in verification (https://github.com/terraform-providers/terraform-provider-azurerm/blob/master/azurerm/helpers/validate/database.go).
  Error: collation contains invalid characters, only underscores are supported, got en-US

• Also current behavior conflicts with our terraform documentation:

collation - (Required) Specifies the Collation for the PostgreSQL Database, which needs to be a valid PostgreSQL Collation. Note that Microsoft uses different notation - en-US instead of en_US. Changing this forces a new resource to be created.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

[ghost]

This has been released in version 1.37.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.37.0"
}
# ... other configuration ...