Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm_storage_account with shared_access_key_enabled and blob restore_policy set fails #24568

Closed
1 task done
ned1313 opened this issue Jan 19, 2024 · 5 comments
Closed
1 task done

Comments

@ned1313
Copy link
Contributor

ned1313 commented Jan 19, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.7.0

AzureRM Provider Version

3.88.0

Affected Resource(s)/Data Source(s)

azurerm_storage_account

Terraform Configuration Files

provider "azurerm" {
  features {

  }
  storage_use_azuread = true
}

# Use random string to create naming suffix
resource "random_string" "main" {
  length  = 6
  special = false
  upper   = false
}

locals {
  naming_string = "abac${random_string.main.result}"
}

# Create resource group
resource "azurerm_resource_group" "main" {
  name     = local.naming_string
  location = "eastus"
}

# Create storage account
resource "azurerm_storage_account" "test" {
  name                = "${local.naming_string}test"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name

  account_tier                      = "Standard"
  account_kind                      = "StorageV2"
  account_replication_type          = "GRS"
  enable_https_traffic_only         = true
  min_tls_version                   = "TLS1_2"
  shared_access_key_enabled         = false
  default_to_oauth_authentication   = true
  infrastructure_encryption_enabled = false


  blob_properties {
    versioning_enabled            = true
    change_feed_enabled           = true
    change_feed_retention_in_days = 90
    last_access_time_enabled      = true

    delete_retention_policy {
      days = 30
    }

    container_delete_retention_policy {
      days = 30
    }

    restore_policy {
      days = 15
    }
  }

}

Debug Output/Panic Output

│ Error: retrieving share properties for Storage Account (Subscription: "XXXXXXXXXXXXXXX"
│ Resource Group Name: "abacnuk2vj"
│ Storage Account Name: "abacnuk2vjtest"): storage.FileServicesClient#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:77c6f886-701a-0060-78f3-4a3998000000\nTime:2024-01-19T16:20:10.8035549Z"
│
│   with azurerm_storage_account.test,
│   on sa_test.tf line 2, in resource "azurerm_storage_account" "test":
│    2: resource "azurerm_storage_account" "test" {

Expected Behaviour

The storage account is created successfully.

Actual Behaviour

The storage account is created in Azure, but errors out in Terraform. Attempts to re-create the resource may fail during planning or apply.

After some additional research, I have determined that the culprit is the combination of the shared_access_key_enabled argument set to true and restore_policy inside the blob_properties block:

restore_policy {
      days = 15
    }

Removing the restore_policy block will allows the storage account to be created successfully.

Two other important things to note:

When Terraform tries to re-create the failed storage account, this change is listed in the execution plan:

- share_properties {
          - retention_policy {
              - days = 7 -> null
            }
        }

No share properties have been set in the configuration.

Also, if I create the storage account without the restore_policy block and then add the block in afterwards, Terraform is able to modify the storage account without error.

~ blob_properties {
            # (4 unchanged attributes hidden)

          ~ restore_policy {
              ~ days = 15 -> 18
            }

            # (2 unchanged blocks hidden)
        }

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
azurerm_storage_account.test: Modifying... [id=/subscriptions/XXXX/resourceGroups/abacnuk2vj/providers/Microsoft.Storage/storageAccounts/abacnuk2vjtest]
azurerm_storage_account.test: Modifications complete after 4s [id=/subscriptions/XXXX/resourceGroups/abacnuk2vj/providers/Microsoft.Storage/storageAccounts/abacnuk2vjtest]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

It only encounters the issue when creating a new storage account.

Looking through the logs, it seems like after the storage account is created, the azurerm provider is trying to get all the properties of the storage account. When it tries to read the properties of the file services, it is trying to use a SAS token from the storage key. Since that's disabled, the request errors in a 404. No idea why it checks the file services when the restore_policy is set, but not otherwise.

2024-01-19T11:31:30.367-0500 [DEBUG] provider.terraform-provider-azurerm_v3.88.0_x5.exe: az-cli invocation: az account get-access-token --scope https://storage.azure.com/.default --tenant f06624a8-558d-45ab-8a87-a88094a3995d -o=json: timestamp=2024-01-19T11:31:30.367-0500
2024-01-19T11:31:31.751-0500 [DEBUG] provider.terraform-provider-azurerm_v3.88.0_x5.exe: AzureRM Request: 
GET /subscriptions/XXXX/resourceGroups/abacnuk2vj/providers/Microsoft.Storage/storageAccounts/abacnuk2vjtest/fileServices/default?api-version=2021-09-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.3 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 storage/2021-09-01 HashiCorp Terraform/1.6.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.88.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: 003f94fe-514c-6397-2380-7be3df0a85e4
Accept-Encoding: gzip: timestamp=2024-01-19T11:31:31.750-0500
2024-01-19T11:31:32.075-0500 [DEBUG] provider.terraform-provider-azurerm_v3.88.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/XXXX/resourceGroups/abacnuk2vj/providers/Microsoft.Storage/storageAccounts/abacnuk2vjtest/fileServices/default?api-version=2021-09-01: 
HTTP/2.0 404 Not Found
Content-Length: 171
Cache-Control: no-cache
Content-Type: application/json
Date: Fri, 19 Jan 2024 16:31:31 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 003f94fe-514c-6397-2380-7be3df0a85e4
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11998
X-Ms-Request-Id: 7444b242-4493-44d7-9791-657ce6380658
X-Ms-Routing-Request-Id: CANADACENTRAL:20240119T163131Z:3c30b51c-a695-44c1-9637-4a2f7ea35bb9

{"error":{"code":"ResourceNotFound","message":"The specified resource does not exist.\nRequestId:28c39305-a01a-003c-3cf4-4a2e94000000\nTime:2024-01-19T16:31:31.5185087Z"}}: timestamp=2024-01-19T11:31:32.075-0500

Steps to Reproduce

  1. Run terraform apply

Important Factoids

No response

References

No response

@magodo
Copy link
Collaborator

magodo commented Jan 22, 2024

@ned1313 I've just tried your above configuration locally, and the apply succeeded for me.

The provider reaches to the file service because the storage account_tier + account_kind supports file service, therefore the provider will access the file service and treat it as a part of this whole resource. As this is a resource manager API, which means it isn't affected by the shared_access_key_enabled.

@ned1313
Copy link
Contributor Author

ned1313 commented Jan 25, 2024

@magodo You're right, I just ran the code I pasted above and it worked without issue. I thought a little more about what I was actually doing, and I realized I had run an apply with a replace of the storage account, so I tried that. BINGO:

$ terraform apply -replace="azurerm_storage_account.main"
...

azurerm_storage_account.main: Destroying... [id=/subscriptions/XXXX/resourceGroups/abac4mhvwe/providers/Microsoft.Storage/storageAccounts/abac4mhvwe]
azurerm_storage_account.main: Destruction complete after 3s
azurerm_storage_account.main: Creating...
azurerm_storage_account.main: Still creating... [10s elapsed]
azurerm_storage_account.main: Still creating... [20s elapsed]

│ Error: retrieving share properties for Storage Account (Subscription: "XXXX"
│ Resource Group Name: "abac4mhvwe"
│ Storage Account Name: "abac4mhvwe"): storage.FileServicesClient#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:2c0aaf14-201a-0040-38a1-4f8e83000000\nTime:2024-01-25T15:16:33.9699991Z"
│
│   with azurerm_storage_account.main,
│   on main.tf line 27, in resource "azurerm_storage_account" "main":
│   27: resource "azurerm_storage_account" "main" {

To recreate the issue, run the following commands with this configuration:

provider "azurerm" {
  features {}
  # Set storage access to use Azure AD instead of storage key SAS
  storage_use_azuread = true
}

data "azurerm_subscription" "main" {}

# Use random string to create naming suffix
resource "random_string" "main" {
  length  = 6
  special = false
  upper   = false
}

locals {
  naming_string = "abac${random_string.main.result}"
}

# Create resource group
resource "azurerm_resource_group" "main" {
  name     = local.naming_string
  location = "eastus"
}

# Create storage account
resource "azurerm_storage_account" "main" {
  name                = local.naming_string
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name

  account_tier                      = "Standard"
  account_kind                      = "StorageV2"
  account_replication_type          = "GRS"
  enable_https_traffic_only         = true
  min_tls_version                   = "TLS1_2"
  shared_access_key_enabled         = true
  default_to_oauth_authentication   = false
  infrastructure_encryption_enabled = false

  sas_policy {
    expiration_period = "00.02:00:00"
    expiration_action = "Log"
  }

}
# Create the storage account
terraform apply -auto-approve

# Replace the storage account
terraform apply -replace="azurerm_storage_account.main"

After you approve the plan, the storage account will be destroyed and recreated, and then will error out as shown above. I don't know why this is happening on replace. The storage account remains tainted in state data, so Terraform will try and replace it on subsequent plans. If you remove the taint manually, then an execution plan comes back with no changes.

This seems to be an issue with eventual consistency in the API. I've removed the blob properties and changed shared_access_key_enabled to true. Based on the logs, it seems like the FileServicesClient is being told the storage account doesn't exist, even though it does.

@magodo
Copy link
Collaborator

magodo commented Jan 26, 2024

This duplicates to #22992, and there is a fix: #23002.

@magodo magodo added bug and removed question labels Jan 26, 2024
@rcskosir
Copy link
Contributor

@ned1313 Thank you for taking the time to open this issue. It looks like this is a duplicate of #22992. We like to try to keep discussions consolidated, so we’re going to close this new issue in favor of #22992 please follow that issue along with the PR #23002. Thanks again!

Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants