Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded #10015

Closed
MykolaPelyp-SoftServe opened this issue Dec 29, 2020 · 20 comments

Comments

@MykolaPelyp-SoftServe
Copy link

Terraform (and AzureRM Provider) Version

Hi, we having that on:
azurerm version = "=2.37.0"
version = "=2.41.0"
version = "=2.30.0"

Terraform Version

terraform is written in Terraform 12 features

v13.5

Affected Resource(s)

azurerm_storage_share

Terraform Configuration Files

resource "azurerm_storage_share" "storage_share" {
  count                = var.file_share_enabled ? 1 : 0
  name                 = var.file_share_name
  storage_account_name = element(azurerm_storage_account.storage_account.*.name, 0)
  quota                = 2048

}

Debug Output

2020/12/29 18:21:53 [TRACE] dag/walk: vertex "module.storage_account (close)" is waiting for "module.storage_account.output.file_share_id (expand)"
2020/12/29 18:21:53 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:53 [TRACE] dag/walk: vertex "root" is waiting for "module.storage_account (close)"
2020/12/29 18:21:55 [TRACE] dag/walk: vertex "module.storage_account.output.file_share_id (expand)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "module.storage_account (close)" is waiting for "module.storage_account.output.file_share_id (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "root" is waiting for "module.storage_account (close)"
2020/12/29 18:22:00 [TRACE] dag/walk: vertex "module.storage_account.output.file_share_id (expand)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:22:03 [ERROR] eval: *terraform.EvalRefresh, err: Error retrieving File Share "thnxthnxfs" (Account "thnxthnxsa" / Resource Group "thnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2020/12/29 18:22:03 [ERROR] eval: *terraform.EvalSequence, err: Error retrieving File Share "thnxthnxfs" (Account "thnxthnxsa" / Resource Group "thnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Expected Behaviour

We expect our state refresh successfully, it is happening on Plan stage, first apply goes without any issues, at the moment we are using -refresh=false as a workaround

Actual Behaviour

Error: Error retrieving File Share "fthnxthnxfs" (Account "fthnxthnxsa" / Resource Group "fthnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

After 10 minutes of refreshing of the state,
We also have 4 other Private Endpoints (after we added them refresh took from ~30 seconds to 5 minutes, after we added fifth Private endpoint for File Share it stopped refreshing and always failing at 10 minutes)

Steps to Reproduce

resource "azurerm_storage_account" "storage_account" {
  count                     = var.enabled ? 1 : 0
  name                      = var.storage_account_name
  resource_group_name       = var.resource_group_name
  location                  = var.location
  account_kind              = var.account_kind
  account_tier              = var.account_tier
  account_replication_type  = var.account_replication_type
  enable_https_traffic_only = var.enable_https_traffic_only
  access_tier               = var.access_tier

  identity {
    type = "SystemAssigned"
  }

  network_rules {
    default_action             = var.network_rules_default_action
    bypass                     = var.network_rules_bypass
    ip_rules                   = var.network_rules_allowed_ip_ranges
    virtual_network_subnet_ids = var.network_rules_allowed_subnet_ids
  }


  tags = merge(
    var.tags,
    {
      "Name" = var.storage_account_name
    },
  )
}
resource "azurerm_storage_share" "storage_share" {
  count                = var.file_share_enabled ? 1 : 0
  name                 = var.file_share_name
  storage_account_name = element(azurerm_storage_account.storage_account.*.name, 0)
  quota                = 2048

} 

module "private_endpoint_sa_file_share" {
    source = "../module-azure-private-endpoint"
    name                            = "thnxthnx"
    location                        = var.location
    resource_group_name             = var.cluster_app_rg
    subnet_id                       = var.cluster_app_subnet_id
    private_endpoint_resource_id    = element(azurerm_storage_account.storage_account.*.id, 0)
    subresource_names               = ["file"]
    private_endpoint_enabled        = var.private_endpoint_enabled
    private_dns_zone_ids            = [module.private_dns_zone_storage_account_file_share.id]
    private_dns_name                = module.private_dns_zone_storage_account_file_share.name
}

module "private_dns_zone_storage_account_file_share" {
    source = "../module-azure-private-dns-zone"
    env                             = var.env
    dns_zone_name                   = "privatelink.file.core.windows.net"
    resource_group_name             = var.cluster_app_rg
    cluster_app_network_id          = var.cluster_app_network_id
}
  1. terraform apply

Notes:
On azurerm providers 2.37 - 2.41 issue description not explaining anything:

Error: shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
@ryan-rockwell

This comment has been minimized.

@cgordoncarroll

This comment has been minimized.

@skobzar

This comment has been minimized.

@MykolaPelyp-SoftServe
Copy link
Author

Hi again,

I am reusing the exact same code on different environment,
There is hard coded version of AzureRM provider =2.30.0 (the same AzureRM provider as on errored env. as we tested most of the code on it)

There is no refresh issue, furthermore, terraform refresh happening in lightning speed without any issues, so main differences between errored environment and working one:

  1. Terraform v0.14.7 (compared to v13.5 on errored)
  2. As for now we have no firewall restrictions established on Storage Account where we store state file

@kiazhi
Copy link

kiazhi commented Mar 8, 2021

Error: Error reading queue properties for AzureRM Storage Account "storageaccountname": queues.Client#GetServiceProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
  • OS: RHEL - Having the issue above

    • terraform v0.14.7
    • registry.terraform.io/hashicorp/azurerm v2.50.0
  • OS: macOS - No issue

    • terraform v0.14.7
    • registry.terraform.io/hashicorp/azurerm v2.50.0

@jrfoster0182
Copy link

Receiving the same issue.

OS: RHEL

  • terraform v0.14.7
  • registry.terraform.io/hashicorp/azurerm v2.36.0

@Chambras
Copy link
Contributor

I am having this issue as well.
OS: 18.04.5 LTS (Bionic Beaver)

  • terraform v0.15.4
  • registry.terraform.io/hashicorp/azurerm v2.60.0

I found out it happens when you add an azurerm_private_endpoint resource for your fileshare.

@kevinsorenson
Copy link

Here too, also on private endpoint for blob storage
OS: Win 10
terraform v0.14.4
registry.terraform.io/hashicorp/azurerm v2.60.0

@RaceFPV
Copy link

RaceFPV commented May 28, 2021

seeing the same here, trying to refresh state on azurerm_storage_container fails
OS: ubuntu
terraform: v0.154
azurerm: 2.61.0

@jhughesjha
Copy link

Having the same issue
OS: Ubuntu
terraform: v1.0.0
azurerm: 2.65.0

@fgarcia-cnb
Copy link

fgarcia-cnb commented Jul 13, 2021

same issue, with azurerm 2.59, rhel and private endpoint. we deploy the same code in westus2 and southcentralus without issue, but we're seeing this in westcentralus. what region is everyone else deploying to?

a bash curl call to the api (https://docs.microsoft.com/en-us/rest/api/storagerp/queue-services/get-service-properties) works fine in all cases

@fgarcia-cnb
Copy link

we figured out our issue. for some reason querying queue properties doesn't go through the azure api, but queries the storage account directly. since we are using private endpoint, we had to open up the nsg to allow port 443 connectivity to the storage account subnet from terraform.

@ronaknnathani
Copy link

Any updates on this issue? Experiencing the same error with terraform v0.14.5, azurerm v2.69.0.

@smark88
Copy link

smark88 commented Sep 1, 2021

Same here I have 443 open and such, no luck

OS="Alpine Linux" (atlantis)
VERSION_ID=3.13.5
TF=0.14.11
azuread=1.6.0
azurerm=2.74.0

The below snippet repeats till 10 minute timeout, exact same as the issue.

2021/09/01 18:44:05 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)"
2021/09/01 18:44:05 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)"
2021/09/01 18:44:06 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
...
2021/09/01 18:47:01 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2021/09/01 18:47:02 [TRACE] vertex "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)": visit complete
2021/09/01 18:47:04 [WARN] Provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new value for module.test201-eu-north-1.module.vault.azurerm_storage_account.main during refresh.
      - .tags: was null, but now cty.MapValEmpty(cty.String)
2021/09/01 18:47:04 [TRACE] EvalWriteState: using RefreshState for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteState: writing current state object for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteDiff: recorded Delete change for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteState: removing state object for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] vertex "module.test201-eu-north-1.module.vault.azurerm_storage_account.main (orphan)": visit complete
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" errored, so skipping
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021/09/01 18:47:04 [INFO] backend/local: plan operation completed
2021-09-01T18:47:04.370Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-01T18:47:04.376Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.74.0/linux_amd64/terraform-provider-azurerm_v2.74.0_x5 pid=200956
2021-09-01T18:47:04.377Z [DEBUG] plugin: plugin exited

@robertbrandso
Copy link
Contributor

I experienced the same as @fgarcia-cnb mentions, the query didn't go through the public Azure API, but connected to the storage accounts private endpoint. Before I opened tcp-443 in Azure Firewall to the private endpoint I got the error message: Error: shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded. After I opened the port everything ran successfully.

I was using the azurerm provider version 2.57.0.

@Tulki
Copy link

Tulki commented Nov 16, 2021

@robertbrandso or @fgarcia-cnb do you happen to have a terraform snippet for how you got this workaround running? Bit confused as to where that's being set, as I hit the same issue even without a firewall rule at the vnet level, and as far as I know according to the docs, NSG rules are ignored for private endpoints with enforce_private_link_endpoint_network_policies = true set on the subnet.

Thank you!

@fgarcia-cnb
Copy link

in our case, its not a tf code issue. its a matter of allowing our VM thats running terraform outbound access to the storage account private endpoint subnet. you are correct, private endpoint doesnt use nsg, but thats on the inbound side.

@robertbrandso
Copy link
Contributor

In the release announcement for azurerm 3.0 it says:

Storage:
We’ll switch to using Resource Manager (management plane) for all resources.

Maybe that'll fix this issue?

@tombuildsstuff
Copy link
Contributor

This'll be fixed by #2977 - please subscribe to that issue for updates

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests