azurerm_backup_protected_file_share Unable to Create Due to Azure Internal Error

[rajakesar]

I am unable to create a backup protected file share using Terraform. I can create it without issue manually. I receive the following error:

Terraform Version

0.12.24

provider "azurerm" {
  # Whilst version is optional, we /strongly recommend/ using it to pin the version of the Provider being used
  version         = "=2.8.0"
  subscription_id = "xxx"
  features {}
}

Affected Resources

azurerm_backup_protected_file_share

Terraform Configuration Files

resource "azurerm_backup_protected_file_share" "share1" {
  resource_group_name       = azurerm_resource_group.rg.name
  recovery_vault_name       = azurerm_recovery_services_vault.vault.name
  source_storage_account_id = azurerm_backup_container_storage_account.sa01_protection.storage_account_id
  source_file_share_name    = var.share_01_name
  backup_policy_id          = azurerm_backup_policy_file_share.policy.id
}

Issue

Error: Azure Backup operation status failed with status "Failed" (Vault "vaultname" Resource Group "vaultrg" Operation ID "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"): Microsoft Azure Backup encountered an internal error.

I reached out the Microsoft for support and received the following:

I see below difference when you enable backup job from Azure portal and Terraform. I suspect Friendly name “input” is causing issue with Terraform to enable backups.

From Azure Portal its using file share unique ID name to enable backup, which is successful.

2020-04-30 15:49:25.0439376 client request id is 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx' 3

2020-04-30 15:49:25.0440571 client app id is 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx' 3

2020-04-30 15:49:25.0441704 CorrelationRequestId is 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx' 3

2020-04-30 15:49:25.0442987 Request Method: PUT, Uri: https://pod01-manag1.cus.backup.windowsazure.com/backupmanagement/resources/******/backupFabrics/Azure/protectionContainers/storagecontainer;storage;stroagerg;fileshare/protectedItems/azurefileshare;xxxxx?api-version=2016-06-01 3

2020-04-30 15:49:25.0444218 the culture info is set to: en-US | Params: {Input Accept Languages = en} 3

Whereas, when you use Terraform, its adding as File share name, which failed.

2020-04-30 17:15:53.9530946 client request id is 'xxxx' 3

2020-04-30 17:15:53.9531809 client app id is 'xxxx' 3

2020-04-30 17:15:53.9532591 CorrelationRequestId is 'xxxx' 3

2020-04-30 17:15:53.9533626 Request Method: PUT, Uri: https://pod01-rrp1.cus.backup.windowsazure.com/Subscriptions/**/resourceGroups/**/providers/Microsoft.RecoveryServices/vaults/***/backupFabrics/Azure/protectionContainers/StorageContainer;***;***;***/protectedItems/AzureFileShare;***?api-version=2019-05-13 3

2020-04-30 17:15:53.9534533 the culture info is set to: en-US | Params: {Input Accept Languages = en-US} 3

I see that the API version has changed from 2016 to 2019. How can I know which API calls TF makes and how can I resolve this issue?

[mybayern1974]

hi @rajakesar , thank you for opening this issue. Could you please provide your Terraform config? And if possible please provide more contexts following this issue template, which could help speed up our troubleshooting. Thanks.

[mepand]

@rajakesar With the recent GA release of Azure File shares backup, a unique identifier has been introduced for each file share and hence for protecting any share, the recommendation is to pass the unique Id of file share in the PUT call rather than friendly name. To fix the issue, you can follow the below steps:

1.Do GET backupprotectableItems API call for the vault that you want to use for configuring protection for file share.
2. Filter the response of above GET call with the friendly name attribute as the "friendly name of your file share".
3. After filtering , the response object will have details only for the file share that you are trying to protect. The name attribute of this response is the unique ID for your file share.
4. Do a PUT protecteditems call using this unique ID rather than friendly name. This should configure protection successfully for your share.

Here are few links for your reference:

https://docs.microsoft.com/en-us/azure/backup/backup-azure-file-share-rest-api#select-the-file-share-you-want-to-back-up

https://docs.microsoft.com/en-us/azure/backup/backup-azure-file-share-rest-api#enable-backup-for-the-file-share

[arbreezy]

I am having the exact same issue under azurerm v2.5.0 and terraform v0.12.8.

@mepand
I guess this uid needs to be introduced in the azurerm provider as well.

Is there anyone looking at this ?

FYI: Adding timeouts doesn't seem to help.

[mepand]

@arbreezy Currently UniqueID is not exposed directly but can be retrieved via API as I mentioned above in my reply. Can you please try the sequence of API calls I mentioned above.

[molinch]

@mepand As you can imagine people using terraform to create resources, are stuck with this kind of breaking change.
I guess we need to wait until next azurerm provider supports this.
And until that happens we basically can't use TF for that.

Wouldn't it have been an option to have it backwards compatible for same time?

[mepand]

@molinch, @arbreezy This change was necessary for the release of an upcoming feature and currently the only way to retrieve Unique ID is via "GET BackupProtectableItems" API call. I understand it would have been easier in case of backward compatibility but if you try the API calls sequence I mentioned above in my reply, configure protection for file shares would work fine. If the same API calls sequence is followed in Terraform script , there won't be any issue.

@arbreezy Can you please elaborate on the timeouts you mentioned in your reply.

[bsuchorowski]

Upvote for this issues. Instruction posted on https://www.terraform.io/docs/providers/azurerm/r/backup_protected_file_share.html
is not working. When do you plan to release a fix for that?

[blugri]

@tombuildsstuff How is this a breaking change if it is currently crashing? I guess in a way it is as it is changing behaviour by no longer crashing. But that is the goal. Currently it is not working.

[katbyte]

@blugri - there was a breaking change on the microsoft service side which caused the resource to stop working, thus it is a breaking change that needs to be fixed.

[mrsimonemms]

There is a possible stop-gap solution by using the Azure CLI - you can use it using the null_resource and then using the local-exec provisioner.

It's not without problems:

• The destroy command will error, so you will have to manually delete the backup first in the Azure portal.
• You need to have az installed and logged in on whatever machine you are running
• It's slow (although I think caused by Azure API)
• Using the local-exec provider is an anti-pattern, or at least a last resort

I would advise against using the command outside Terraform (which was my initial thought). This is because if you rerun this command, it will fail. Therefore, running it in Terraform and appending it to the state makes a lot of sense.

However, as soon as this issue is fixed, I would strongly recommend changing it back to the officially supported resource.

resource "azurerm_storage_account" "account" {
  location = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  name = "example"
}

resource "azurerm_storage_share" "data" {
  name = "data-accunt"
  storage_account_name = azurerm_storage_account.account.name
  quota = 50
}

resource "null_resource" "enable-backup" {
  provisioner "local-exec" {
    command = <<EOT
az backup protection enable-for-azurefileshare \
  --resource-group "${azurerm_resource_group.rg.name}" \
  --vault-name "${azurerm_recovery_services_vault.vault.name}" \
  --azure-file-share "${azurerm_storage_share.data.name}" \
  --policy-name "${azurerm_backup_policy_file_share.policy.name}" \
  --storage-account "${azurerm_storage_account.account.name}"
EOT
  }
}

There may be other configurations that work, but this works for me. For the record, I'm doing this on a Data Lake Gen2 account, but this also seems to work on normal storage accounts although I've done much less extensive testing on that.

[seboss666]

For an unkown reason I did not find this issue before creating a new case at Microsoft Support. I encountered the same problem with terraform 0.12.29 and azurerm provider 2.31.1.

Here what they told me :

I have some news from the Product Team:

In Terraform template, customers use friendly name of file share to configure backup which is currently not supported. However, for providing ARM template support, we have recently updated our APIs to allow configure backup using friendly names.
This functionality would be available soon.

Terraform Team would need SDK upgrade to start consuming the new APIs.

Indeed, there are updates to API versions that are still not used.

[maustin-iwm]

Using azurerm 2.41.0 I am able to add a storage account file share to the storage container for the backup to run, however Terraform reports an error back from Azure saying that the process has failed:

Error creating/udpating Azure File Share backup item "AzureFileShare;abc123def456" (Vault "my-vault-name"): backup.ProtectedItemsClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="BMSUserErrorDataSourceObjectNotFound" Message="Item not found.\r\nSelected item might have been deleted. Please check if item is registered to the vault."

[nkiraly]

The 404s during post creation ID look up when using 2.41.0 are fixed by #9015.
I do believe when that fix merges this issue can be closed as all mentioned problems will have been fixed in master.

[kibnelbachyr]

@nkiraly updating to latest versions of the azurerm provider (2.41.0, 2.42.0) didn't fix my issue with those 404 error during post creation.

[maustin-iwm]

Using version 2.43 the problems seem to continue with 404's during post creation.

[bieksaz]

Can confirm 2.46.1 works fine

[tombuildsstuff]

Closing this out as it sounds like this is now fixed

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!