Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm_storage_account - prevent service endpoint error 404 durring re-create #23002

Closed

Conversation

petr-stupka
Copy link
Contributor

@petr-stupka petr-stupka commented Aug 17, 2023

resolve #22992
resolve #19055
resolve #18897
resolve #15609
resolve #12627

May be solution for #20257

Added loop for reading blob, file, queue and static website properties with timeout 5 min for each service

log level INFO:

Plan: 1 to add, 0 to change, 0 to destroy.
2023-08-18T00:07:18.509+0200 [INFO]  provider: configuring client automatic mTLS
2023-08-18T00:07:18.532+0200 [INFO]  provider.terraform-provider-azurerm: configuring server automatic mTLS: timestamp=2023-08-18T00:07:18.532+0200
azurerm_storage_account.test: Creating...
azurerm_storage_account.test: Still creating... [10s elapsed]
azurerm_storage_account.test: Still creating... [20s elapsed]
2023-08-18T00:07:50.600+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:b0293f99-401e-0072-3f57-d110eb000000\nTime:2023-08-17T22:07:50.6104106Z": timestamp=2023-08-18T00:07:50.600+0200
2023-08-18T00:07:55.822+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:843d7103-601e-004a-5857-d1b42b000000\nTime:2023-08-17T22:07:55.8326890Z": timestamp=2023-08-18T00:07:55.822+0200
azurerm_storage_account.test: Still creating... [30s elapsed]
2023-08-18T00:08:01.040+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:caa0a015-a01e-009e-1357-d1047a000000\nTime:2023-08-17T22:08:01.0526371Z": timestamp=2023-08-18T00:08:01.040+0200
2023-08-18T00:08:06.281+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:4399273b-001e-0074-1357-d1d71a000000\nTime:2023-08-17T22:08:06.2927747Z": timestamp=2023-08-18T00:08:06.281+0200
azurerm_storage_account.test: Still creating... [40s elapsed]
2023-08-18T00:08:11.503+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:0b527d09-701e-0041-6857-d1bb0e000000\nTime:2023-08-17T22:08:11.5119259Z": timestamp=2023-08-18T00:08:11.503+0200
2023-08-18T00:08:16.716+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:ff9f3cd8-c01e-007b-0a57-d1a176000000\nTime:2023-08-17T22:08:16.7411096Z": timestamp=2023-08-18T00:08:16.715+0200
azurerm_storage_account.test: Still creating... [50s elapsed]
2023-08-18T00:08:21.953+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:72e2f39d-b01e-0031-6d57-d1864b000000\nTime:2023-08-17T22:08:21.9672107Z": timestamp=2023-08-18T00:08:21.952+0200
azurerm_storage_account.test: Still creating... [1m0s elapsed]
2023-08-18T00:08:27.185+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:36183d5e-701e-0001-3f57-d13884000000\nTime:2023-08-17T22:08:27.1972534Z": timestamp=2023-08-18T00:08:27.185+0200
2023-08-18T00:08:32.393+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:fd18ddec-e01e-0061-1557-d1441b000000\nTime:2023-08-17T22:08:32.4046359Z": timestamp=2023-08-18T00:08:32.392+0200
azurerm_storage_account.test: Still creating... [1m10s elapsed]
2023-08-18T00:08:37.621+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:93b2abba-801e-0052-1657-d16b4c000000\nTime:2023-08-17T22:08:37.6313915Z": timestamp=2023-08-18T00:08:37.620+0200
2023-08-18T00:08:42.855+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:e089e35e-b01e-0059-7157-d19027000000\nTime:2023-08-17T22:08:42.8631422Z": timestamp=2023-08-18T00:08:42.855+0200
azurerm_storage_account.test: Still creating... [1m20s elapsed]
2023-08-18T00:08:48.069+0200 [INFO]  provider.terraform-provider-azurerm: refreshing static website properties Storage Account (Subscription: "d8714754-66ff-4440-8712-570f2c86ef2b"
Resource Group Name: "azurerm-resources"
Storage Account Name: "storageaccountnamex"): accounts.Client#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:f0f1df6d-f01e-0058-7557-d1cffb000000\nTime:2023-08-17T22:08:48.0801024Z": timestamp=2023-08-18T00:08:48.069+0200
azurerm_storage_account.test: Creation complete after 1m26s [id=/subscriptions/d8714754-66ff-4440-8712-570f2c86ef2b/resourceGroups/azurerm-resources/providers/Microsoft.Storage/storageAccounts/storageaccountnamex]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

@petr-stupka petr-stupka marked this pull request as draft August 17, 2023 21:55
@petr-stupka petr-stupka marked this pull request as ready for review August 17, 2023 22:12
@petr-stupka petr-stupka changed the title azurerm_storage_account - prevent service endpoint error 404 durring re-create azurerm_storage_account - prevent service endpoint error 404 durring re-create Aug 18, 2023
@petr-stupka
Copy link
Contributor Author

Hi @magodo if you would like to format the INFO log messages, please let me know. Thank you

@magodo
Copy link
Collaborator

magodo commented Aug 22, 2023

Hi @petr-stupka, thank you for this PR! For the retrying logic, can we use the pluginsdk.StateChangeConf instead of hardcoding those values, an example can be found at

stateConf := &pluginsdk.StateChangeConf{
Pending: []string{"Exists"},
Target: []string{"NotFound"},
Refresh: dedicatedHostDeletedRefreshFunc(ctx, client, *id),
MinTimeout: 10 * time.Second,
ContinuousTargetOccurence: 20,
Timeout: d.Timeout(pluginsdk.TimeoutDelete),
}

@magodo magodo added the bug label Aug 22, 2023
@petr-stupka
Copy link
Contributor Author

petr-stupka commented Aug 22, 2023

Hi @magodo, pluginsdk.StateChangeConf make sense. I wasn't aware about it.

In one of my test, two endpoints took time to get available, so that looks good

image

@petr-stupka petr-stupka requested a review from magodo August 23, 2023 13:33
Copy link
Collaborator

@WodansSon WodansSon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@petr-stupka, this is looking pretty good so far. I left a few comments if you can take a look and let me know what you think. 🚀

internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@WodansSon WodansSon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @petr-stupka, thanks for pushing those changes. I have left a few more comments and I don't think that the resourceStorageAccountRead function is the correct place for doing these waits. I think it would be much better if you could relocate that logic to the bottom of resourceStorageAccountCreate and the resourceStorageAccountUpdate functions, if it makes sense to put it in the update function.

internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
@WodansSon
Copy link
Collaborator

WodansSon commented Sep 21, 2023

Hi @petr-stupka, no worries... we have not forgotten about this we are currently discussing the PR internally and will address it shortly, sorry for the delay or lack of communication to that effect, but we are looing at this... 🙂

Copy link
Contributor

@tombuildsstuff tombuildsstuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @petr-stupka

Thanks for pushing those changes - I've taken a look through and left some comments inline, but if we can fix those up then we should be able to take another look at get this one in 👍

Thanks!

internal/services/storage/storage_account_resource.go Outdated Show resolved Hide resolved
Refresh: func() (interface{}, string, error) {
properties, err := queueClient.GetServiceProperties(ctx, id.ResourceGroupName, id.StorageAccountName)
if err != nil {
return handleStorageServiceError("queue", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can simplify this to:

Suggested change
return handleStorageServiceError("queue", err)
return "Error", "Error", fmt.Errorf("waiting for the Blob %s to become available: %+v", id, err")

Since the handleStorageServiceError function below assumes that this is Completed if it's not a 404 (which would mean we mark this as Completed when it's an error?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the queueClient.GetServiceProperties function returns error (this may be the 404 error or any other error storage API may return)

This error (generic error type) is handled by handleStorageServiceError function. It return Pending, in case of 404, however any other error is returned imidiately with Completed to brake the loop.

This has been requested by @magodo to avoid consuming all errors.

If i understand the logic correctly, in your example the loop will continue also in case of non 404 error?

And one exception here for queue endpoint > there is no error in case of 404 and therefore i put there the workaround below if properties == nil

You the owners, you can say the final word how the code should looks like. From my point of view the function handleStorageServiceError make sense and therefore asking for confirmation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tombuildsstuff Could you please review above comment?

Refresh: func() (interface{}, string, error) {
_, err = fileServiceClient.GetServiceProperties(ctx, id.ResourceGroupName, id.StorageAccountName)
if err != nil {
return handleStorageServiceError("file", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(as above) we should be able to just return Pending here instead?

Refresh: func() (interface{}, string, error) {
_, err = accountsClient.GetServiceProperties(ctx, id.StorageAccountName)
if err != nil {
return handleStorageServiceError("static website", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(as above) we should be able to just return Pending here instead?

Comment on lines +2426 to +2441
func handleStorageServiceError(service string, err error) (interface{}, string, error) {
// if the error is a DetailedError type, check the status code and return the appropriate state
var respErr azautorest.DetailedError
if errors.As(err, &respErr) {
log.Printf("[DEBUG] error fetching %s service properties: statusCode=%d, message=%s, error=%s\n", service, respErr.StatusCode.(int), respErr.Message, err)
// if the status code is 404 (not found), retry the request
if respErr.StatusCode.(int) == http.StatusNotFound {
// if the status code is 404 (not found), retry the request
log.Printf("[DEBUG] %s service is not available, retrying...\n", service)
return false, "Pending", nil
}
}
// if the error is unhandled type or status, log the error and return the error state
log.Printf("[DEBUG] unexpected error while fetching %s service properties: %v\n", service, err)
return nil, "Completed", err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as such I think this function can be removed?

Comment on lines 115 to 118
{
// destroy the account and recreate it
Config: r.recreate(data),
Destroy: true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would destroy the resource group too, instead we can delete just the account via:

Suggested change
{
// destroy the account and recreate it
Config: r.recreate(data),
Destroy: true,
{
// destroy the account so that we can recreate it below
Config: r.template(data),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed. also for basic account

@petr-stupka
Copy link
Contributor Author

petr-stupka commented Oct 11, 2023

Hi @tombuildsstuff

sorry for the delay, i reviewed your suggested changes, i would like to ask for confirmation the error handling is not required. See #23002 (comment)

@WodansSon
Copy link
Collaborator

@petr-stupka, I have brought this up in our sync, we will get to it asap 🙂

@petr-stupka
Copy link
Contributor Author

Hi @WodansSon just a kindly reminder

@itdevops-channelcapital
Copy link

itdevops-channelcapital commented Nov 30, 2023

@petr-stupka Has there been any further progress in getting this PR merged?
@WodansSon

@petr-stupka
Copy link
Contributor Author

Hi guys @magodo @WodansSon can you please let me know how to proceed with the PR? Should i modify it like @tombuildsstuff proposed please?

@petr-stupka
Copy link
Contributor Author

Hi @magodo @WodansSon do you think we can make a progress on this? AS i said i can do what Tom requested or adjust it with the error handling described above

@magodo
Copy link
Collaborator

magodo commented Mar 6, 2024

@petr-stupka I've ping tom and hopefully he'll look at that comment when he has bandwidth.

Copy link

github-actions bot commented Apr 8, 2024

This PR is being labeled as "stale" because it has not been updated for 30 or more days.

If this PR is still valid, please remove the "stale" label. If this PR is blocked, please add it to the "Blocked" milestone.

If you need some help completing this PR, please leave a comment letting us know. Thank you!

@tombuildsstuff
Copy link
Contributor

hey @petr-stupka

Thanks for this PR and apologies for the delayed review/response here.

There’s been a number of changes to the Storage resources since this PR was first opened, namely the Data Plane SDK and it’s base layer have been updated to make use of hashicorp/go-azure-sdk’s base layer rather than Azure/go-autorest (in addition to some other changes to the resource).

Since this PR is currently relying on the error behaviour of the older base layer, the logic in this PR would need adapting to account for that - however there’s also some larger changes needed for the Storage Account resource as a part of moving it to hashicorp/go-azure-sdk.

I’ve recently been looking into porting the remaining Storage resources over to using hashicorp/go-azure-sdk instead of Azure/azure-sdk-for-go and have opened #26218 which does this.

Whilst I took a look into updating this PR to account for that work, when I pulled this down locally to test/reproduce this, I realised the test case doesn’t recreate the Storage Account at present and as such it would also require updating.

As such I ended up including a fix for #22992 in #26218 - which both updates this logic to use the new base layer (and the custompollers that we’re now using, which are what hashicorp/go-azure-sdk exposes) - and updates the test to ensure that we’re deleting and recreating the same Storage Account in the same region.

As such whilst I’d like to thank you for this contribution and apologise that this PR has been sitting for a while - I hope you don’t mind but I’m going to close this PR in favour of #26218 which incorporates a fix for #22992.

Thanks!

@petr-stupka
Copy link
Contributor Author

petr-stupka commented Jun 5, 2024

Cool, i don't mind closing this PR, important is the bug has been fixed! Thank you for the update!

@petr-stupka petr-stupka deleted the b/storage-account-error-404 branch June 5, 2024 14:45
Copy link

github-actions bot commented Jul 6, 2024

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.