azurerm_storage_share - enabling firewall on storage account makes the request not authorized

[ryanps1]

After upgrading to 1.32.0 refreshing an existing storage share fails with the following error:

Error retrieving File Share "XYZ" (Account "XYZ" / Resource Group "RG"): shares.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.\nRequestId:xxx\nTime:xxx</Message></Error>" error: invalid character 'ï' looking for beginning of value.

To reproduce the issue, apply the following config:

resource "azurerm_resource_group" "test" {
  name     = "azuretest"
  location = "West Europe"
}

resource "azurerm_storage_account" "test" {
  name                     = "azureteststorage"
  resource_group_name      = "${azurerm_resource_group.test.name}"
  location                 = "${azurerm_resource_group.test.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_share" "testshare" {
  name                 = "sharename"
  storage_account_name = "${azurerm_storage_account.test.name}"
  quota                = 50
}

After the initial terraform apply, when you attempt to do another plan it will fail with the above error.

Terraform v0.11.11

[borikoss]

Similar issue got in the deployment (Terraform v.0.11.13).

Fallback to the previous Azurerm Provider v.31.0 helps. Error message:

* module.storageacountcontainer_data_premium.azurerm_storage_container.diskstorage: azurerm_storage_container.diskstorage: Error retrieving Container "vhd" (Account "<removed>" / Resource Group "<removed>"): containers.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:ceb74949-b01c-0003-57d0-424cca000000\nTime:2019-07-25T10:04:19.1120420Z</Message></Error>" error: invalid character 'ï' looking for beginning of value[0m

[katbyte]

Hi @ryanps1 & @borikoss,

So i have tried to repo this locally to no avail:

[12:26:03] kt@snowbook:~/hashi/tf/gh/az-3925$ cat main.tf
resource "azurerm_resource_group" "test" {
  name     = "kttest-20190725-az3925"
  location = "West Europe"
}

resource "azurerm_storage_account" "test" {
  name                     = "kttest20190725"
  resource_group_name      = "${azurerm_resource_group.test.name}"
  location                 = "${azurerm_resource_group.test.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_share" "testshare" {
  name                 = "ktsharename"
  storage_account_name = "${azurerm_storage_account.test.name}"
  quota                = 50
}
[12:26:05] kt@snowbook:~/hashi/tf/gh/az-3925$ tfinitautoapply

Initializing the backend...

Initializing provider plugins...

...

* provider.azurerm: version = "~> 1.32"

Terraform has been successfully initialized!

...

azurerm_resource_group.test: Creating...
azurerm_resource_group.test: Creation complete after 4s [id=/subscriptions/x/resourceGroups/kttest-20190725-az3925]
azurerm_storage_account.test: Creating...
azurerm_storage_account.test: Still creating... [10s elapsed]
azurerm_storage_account.test: Still creating... [20s elapsed]
azurerm_storage_account.test: Creation complete after 27s [id=/subscriptions/x/resourceGroups/kttest-20190725-az3925/providers/Microsoft.Storage/storageAccounts/kttest20190725]
azurerm_storage_share.testshare: Creating...
azurerm_storage_share.testshare: Creation complete after 4s [id=https://kttest20190725.file.core.windows.net/ktsharename]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
[12:26:52] kt@snowbook:~/hashi/tf/gh/az-3925$

And a quick google leads me to this: https://stackoverflow.com/questions/31398044/got-error-invalid-character-%C3%AF-looking-for-beginning-of-value-from-json-unmar however I am unsure why you are getting this.

Could you provide some more details of your environment/how you are accessing azure that might be different? A debug log would go a long way too.

Thanks,
kt

[ryanps1]

Hi @katbyte - Could you attempt another plan after the resources have been added? I don't have any issues with the initial plan and apply, but it fails on any subsequent plans.

Happy to provide a more in-depth report/any debug logs that may be of use otherwise.

Thanks,
Ryan

[katbyte]

@ryanps1,

Both a tf plan and tf apply completed without error 🙁

[ryanps1]

Hi,

I've sent an email with the debug logs attached.

For reference, I'm using the remote backend for AzureRM and authenticating with a service principal defined as environment variables. Would be curious to see if @borikoss is in a similar position?

Thanks

[borikoss]

@ryanps1

Yes, we are using both ARM_ACCESS_KEY and TF_VAR_azure_client_secret as environment variables.
Yes, we are storing the TF statefile remotely in a Azure Blob.

thx

[ryanps1]

Update:

This only happens when the storage account in question has a firewall enabled – I don’t manage the firewall rules within Terraform, so I hadn’t considered this as much as I should have.

The pipeline seems to work absolutely fine when the storage account has no firewall enabled with both 1.31.0 and 1.32.0. However, as soon as the firewall is enabled (even though the source IP address where Terraform is operating from is whitelisted) it fails with the following errors:

The error in 1.31.0:
* module.StorageAccountShare.azurerm_storage_share.storageshare: azurerm_storage_share.storageshare: Error testing existence of share "XYZ": storage: service returned error: StatusCode=403, ErrorCode=403 This request is not authorized to perform this operation., ErrorMessage=no response body was available for error status code, RequestInitiated=Fri, 26 Jul 2019 10:24:20 GMT, RequestId=xxx, API Version=, QueryParameterName=, QueryParameterValue=

The error in 1.32.0:
Error retrieving File Share "XYZ" (Account "XYZ" / Resource Group "RG"): shares.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.\nRequestId:xxx\nTime:xxx</Message></Error>" error: invalid character 'ï' looking for beginning of value.

I’m not sure if there’s been a change that’s now impacting AzureRM that will need to be reflected, or if something has changed in Azure. I'll follow up elsewhere for Azure too.

Thanks

[borikoss]

I am not sure if this a firewall issue.
I am deploying with 1.31.0 version without error. But from the same machine, under same conditions there is an error with 1.32.0. The same error on 1.32.0 happens also not only from local deployment ( PC -> Network -> Azure), but also on Jenkins which is located inside of Azure network

[katbyte]

@ryanps1,

Thanks for the update, given it also happens in 1.31 i am going to update the issue to reflect that.

Sounds like something has changed on the azure side wrt firewalls if 1.31 used to work. (and the newer version of autorest we are using is having issues decoding the response.)

[katbyte]

@borikoss, Could you share a tf configuration that fails in 1.32 and not 1.31?

[borikoss]

@katbyte it is the same configuration which works on 1.31 and fails on 1.32. I can not share the deployment since we have on customer side more then 300 ressources including multiple var-files und multiple modules. Which ressouces would you need to check?

[katbyte]

@borikoss, I posted a TF config above, does that one fail on either?

[bottkars]

same here when using 1.32, container creation fails with above bug on Premium Storage Accounts, 1.31 works as supposed

[borikoss]

@borikoss, I posted a TF config above, does that one fail on either?

@katbyte
the code above #3925 (comment)
works fine for us. Can plan & apply with 1.32

[ryanps1]

Hey @katbyte - would you be able to provide any ETA on this?

Thanks in advance

[invidian]

I'm currently bisecting between v1.31.0 and v.1.32.0 and after some analysis, I think there are 2 bugs out there. Outer which break decoding error message and inner which causes the unknown error message. So far I was able to get following error message, which I haven't seen before:

1 error occurred:
	* azurerm_storage_container.blob_container: 1 error occurred:
	* azurerm_storage_container.blob_container: Error creating container "bug3939-container-mateusz24" in storage account "bug3939testmemateusz23": storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2c461284-501e-0022-71fa-5738ad000000
Time:2019-08-21T08:32:28.7784665Z, RequestInitiated=Wed, 21 Aug 2019 08:32:28 GMT, RequestId=2c461284-501e-0022-71fa-5738ad000000, API Version=, QueryParameterName=, QueryParameterValue=

Which response code (403) matches the responses we've seen before.

[invidian]

Just finished bisecting the decoding error and it seems the first bad commit is: 2cfca67.

Used following to reproduce:

variable "location" {
  default = "westeurope"
}

provider "azurerm" {
  version = "~> 1.32"
}

resource "azurerm_resource_group" "rg" {
  name = "mateusz-azurerm-bug3939-test-newiwest"
  location = "${var.location}"
}

resource "azurerm_storage_account" "sa" {
  name                      = "bug3939testmemateusz23"
  resource_group_name       = "${azurerm_resource_group.rg.name}"
  location                  = "${var.location}"
  account_kind              = "StorageV2"
  account_tier              = "Standard"
  account_replication_type  = "LRS"
  enable_https_traffic_only = "true"
}

resource "azurerm_storage_container" "blob_container" {
  name                  = "bug3939-container-mateusz24"
  storage_account_name  = "${azurerm_storage_account.sa.name}"
  container_access_type = "blob"
  resource_group_name       = "${azurerm_resource_group.rg.name}"
}

Then run Terraform with this loop: terraform init; while true; do terraform taint azurerm_storage_container.blob_container; terraform taint azurerm_storage_account.sa; terraform apply -auto-approve || break; sleep 1; done.

[ghost]

This has been released in version 1.33.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.33.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!