Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers #26532

snemetz · 2020-10-09T16:48:36Z

Terraform Version

Terraform v0.13.4
Also tried 0.13.3 got same issue

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"
  required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

Expected Behavior

Expected init to work

Actual Behavior

Initializing provider plugins...

Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

Steps to Reproduce

terraform init

Additional Context

I can use curl and access the url that terraform says it can't access with no issues
I've tried multiple times over 2 days and issue doesn't change. Terraform always fails, curl always succeeds
Had no issues with 0.12, but 0.13 has not worked yet
Terraform section was the only change made going to 0.13
This is on Mac
I also tried with only the posted terraform config and still get same issue

The text was updated successfully, but these errors were encountered:

crosbymichael1 · 2020-10-13T06:14:18Z

Im having the same issue trying to use the helm provider:

terraform {
  required_providers {
    helm = {
      source = "hashicorp/helm"
      version = "1.3.2"
    }
  }
}

provider "helm" {

  kubernetes {
    host                   = local.kube_host
    cluster_ca_certificate = local.kube_cluster_ca_certificate_decoded
    token                  = local.kube_token
    load_config_file       = false
  }
}

Error Message:

`Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

Finding latest version of hashicorp/aws...
Finding hashicorp/helm versions matching "1.3.2"...
Using hashicorp/aws v3.9.0 from the shared cache directory

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/helm:
no available releases match the given constraints 1.3.2
`

pselle · 2020-10-14T20:25:03Z

Thank you for the report! Short of more information to help replicate this issue, I've tagged this as a registry issue as perhaps the concern may lie there.

@crosbymichael1 -- it looks like your message cut of at the error message, so I'm not sure you have the same error here (the timeout).

crosbymichael1 · 2020-10-14T20:35:59Z

@pselle I figured out what was going on

so I did a tg apply in one of the helm folders and got this error, which I thought was pretty weird because there is 100% a

helm provider version > 1.3
Initializing provider plugins...

Finding hashicorp/helm versions matching "~> 1.3"...
Finding latest version of hashicorp/aws...
Using hashicorp/aws v3.9.0 from the shared cache directory
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider hashicorp/helm:
no available releases match the given constraints ~> 1.3

After changing the version a bunch of times I found out that since I already had a different version of the binary in my tf plugins folder, that instead of downloading the new version, terraform just checked locally to see what I had installed, and since only 10.5 is there, well that is the only version I'd be allowed to use
/Users/michael.crosby/.terraform.d/plugins/registry.terraform.io/hashicorp/helm/10.5
So I just deleted my local helm provider folder and then did a tg apply and it downloaded the version I specified and it worked fine then

pselle · 2020-10-14T20:56:42Z

Thanks for the additional information @crosbymichael1! I've re-added the new label here because that gives more information that might help someone create a reproduction case for this particular angle of the issue (which could be separate from the original post, unsure, but what you've described does seem problematic, and tricky to detangle!)

crosbymichael1 · 2020-10-14T21:05:31Z

@snemetz Try to delete your local aws file in the terraform plugins folder and then rerun terraform init and see if it resolves your issue:

rm -rf ~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws

If your changing aws provider version I have a feeling if terraform see you already have a provider installed locally then its only checking for versions within your local folder and not out in the terraform registry.

snemetz · 2020-10-15T02:46:01Z

That dir was empty. plugin-cache had a lot of versions. Remove all
` ls -lR
total 8
-rw-r--r-- 1 stevennemetz staff 394 Nov 16 2018 checkpoint_signature
drwxr-xr-x 3 stevennemetz staff 96 Jul 1 2019 plugin-cache
drwxr-xr-x 3 stevennemetz staff 96 Jun 21 2019 plugins

./plugin-cache:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Oct 14 19:37 darwin_amd64

./plugin-cache/darwin_amd64:

./plugins:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Nov 5 2019 darwin_amd64

./plugins/darwin_amd64:`

Made no change. Same error.
The error claims it is about downloading the versions file. Which should be before comparing with local files.

snemetz · 2020-10-15T02:53:30Z

Debug log
`
2020/10/14 19:49:59 [INFO] Terraform version: 0.13.4
2020/10/14 19:49:59 [INFO] Go runtime version: go1.14.7
2020/10/14 19:49:59 [INFO] CLI args: []string{"/usr/local/bin/terraform13.4", "init"}
2020/10/14 19:49:59 [DEBUG] Attempting to open CLI config file: /Users/stevennemetz/.terraformrc
2020/10/14 19:49:59 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] will search for provider plugins in /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: /Users/stevennemetz/.terraform.d/plugins is a symlink to /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Users/stevennemetz/Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [INFO] CLI command args: []string{"init"}
2020/10/14 19:49:59 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2020/10/14 19:49:59 [DEBUG] New state was assigned lineage "d727669d-a920-07ec-4a64-bc51e80e8402"
2020/10/14 19:49:59 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2020/10/14 19:49:59 [TRACE] Meta.Backend: instantiated backend of type
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "."
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend does not support operations, so wrapping it in a local backend
2020/10/14 19:49:59 [TRACE] backend/local: state manager for workspace "default" will:

read initial snapshot from terraform.tfstate
write new snapshots to terraform.tfstate
create any backup at terraform.tfstate.backup
2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: read nil snapshot
2020/10/14 19:49:59 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
2020/10/14 19:49:59 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/14 19:49:59 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/14 19:50:02 [ERR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=darwin&signature=069ea326-f2fa-49a8-980f-5a8c5830d0cb&version=0.13.4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/10/14 19:50:10 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/14 19:50:10 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/14 19:50:20 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/10/14 19:50:20 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/14 19:50:21 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/14 19:50:21 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/14 19:50:31 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
`

techdragon · 2020-10-16T17:11:23Z

I have this issue as well, its been happening since I switched to using any 0.13.x version. To me it looks like the new provider resolution functionality doesn't function correctly with having set a plugin_cache_dir setting.

techdragon · 2020-10-16T17:16:52Z

This may actually be a case of #26223

Edit: Yep, I just fixed my case of this issue with mv ~/.terraform.d/plugins ~/.terraform.d/plugin-cache and updating my .terraformrc file from plugin_cache_dir = "$HOME/.terraform.d/plugins" to plugin_cache_dir = "$HOME/.terraform.d/plugin-cache" problem solved instantly with all existing cached providers still recognised.

@snemetz Does this work for you?

C-Howard · 2020-10-19T14:45:50Z

I can't speak for snemetz but I appear to be seeing the same issue and I have no files in ~/.terraform.d/ besides a checkpoint_signature file

mildwonkey · 2020-10-19T16:11:48Z

It looks like there are two different problems being reported in this issue: a registry timeout error for the AWS provider, and some provider installer-specific questions and issues. I'll see how many I can address!

Error: Failed to query available provider packages w/net/http: request canceled while waiting for connection

Unfortunately I am not sure what caused this - I can't reproduce the issue now, and it's suspicious that curl was working for you while init was failing. I see that the same error messages were reported on two different days; do any of you still have this still problem today? I will check with our internal registry team and see if they have any ideas or information.

Terraform not updating provider version

Terraform v0.13 will not query the registry for any provider it finds locally installed under ${CONFIG_DIR}.terraform/plugins. Run terraform init -upgrade to tell terraform to query the registry for provider versions and install any updates meeting your version constraints.

This mechanism is changing in v0.14 so there is a separate installation directory for locally installed (non-registry) providers; I'm afraid I will mangle the details but you can take a look at the (draft!!) of the v0.14 upgrade guide for a nice explanation of the new options: https://github.com/hashicorp/terraform/blob/master/website/upgrade-guides/0-14.html.markdown#the-local-provider-cache-directory

The TL;DR is that we recommend you use the filesystem_mirror configuration for local providers, but in v0.14 you will also be able to drop them in the following directory (relative to the terraform configuration files): terraform.d/plugins/ . It's important that we can designate directories that only terraform is allowed to write to, and will make sure the documentation is clear and add warnings where appropriate so it's clear to y'all.

Issues when setting plugin_cache_dir to implicit local mirror directory #26223

Terraform expects to write to the plugin_cache_dir and so it is invalid as a local mirror. We'll take a look at improving the documentation and printing a warning or error message if the provider installer finds problematic configuration like this. I'd like to keep further conversation about this specifically in #26223 , please.

mildwonkey · 2020-10-19T16:42:04Z

Quick update: the registry team dug into their logs based on the timestamp on the original reported issue, and did not see anything suspicious, nor were there any reported outages with our various service providers. At this point I can't offer any explanation for the timeout beyond "vague unspecified network hiccups", but we'll definitely keep our eyes open and dig in if this happens again.

C-Howard · 2020-10-19T16:43:56Z

I am currently verifying and seeing the issue consistently.

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 2.70.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

My versions.tf file for the module seeing the error is the following

terraform {
  required_providers {
    aws = {
     source  = "hashicorp/aws"
     version = "~> 2.70.0"
    }
  }
  required_version = ">= 0.13"
}

This looks to be the same scenario as the original reported issue. We use no in-house providers, only the hashicorp AWS provider.

mildwonkey · 2020-10-19T16:56:52Z

@C-Howard thanks! Is that happening right now ? I'll report this to the registry folks; it will help if you can share your TRACE logs

C-Howard · 2020-10-19T17:02:36Z

Yes it happens every time I run terraform init.

[terragrunt] [[redacted]/account] 2020/10/19 12:59:11 Running command: terraform --version
[terragrunt] 2020/10/19 12:59:15 Terraform version: 0.13.4
[terragrunt] 2020/10/19 12:59:15 Reading Terragrunt config file at [redacted]/account/terragrunt.hcl
[terragrunt] 2020/10/19 12:59:23 Running command: terraform init -backend-config=bucket=[redacted] -backend-config=dynamodb_table=[redacted] -backend-config=encrypt=true -backend-config=key=[redacted] -backend-config=region=us-east-1 -upgrade
2020/10/19 12:59:23 [INFO] Terraform version: 0.13.4  
2020/10/19 12:59:23 [INFO] Go runtime version: go1.14.7
2020/10/19 12:59:23 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init", "-backend-config=bucket=[redacted]", "-backend-config=dynamodb_table=[redacted]", "-backend-config=encrypt=true", "-backend-config=key=[redacted]/terraform.tfstate", "-backend-config=region=[redacted]", "-upgrade"}
2020/10/19 12:59:23 [DEBUG] Attempting to open CLI config file: /Users/[redacted]/.terraformrc
2020/10/19 12:59:23 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/.terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/Library/Application Support/io.terraform/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins

Initializing provider plugins...
2020/10/19 13:00:16 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/10/19 13:00:16 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
- Finding hashicorp/aws versions matching "~> 2.70.0"...
2020/10/19 13:00:16 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:16 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:26 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:26 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:36 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded
2020/10/19 13:00:36 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/19 13:00:37 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/19 13:00:37 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:48 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context
deadline exceeded (Client.Timeout exceeded while awaiting headers)

[terragrunt] 2020/10/19 13:00:48 Hit multiple errors:
exit status 1

mildwonkey · 2020-10-19T17:21:45Z

Thanks, I appreciate the quick response - we'll dig into this on our end.

For anyone else reading this issue: Any new information is appreciated; anyone else experiencing this same issue please reply to the original comment with a 👍 (instead of adding comments that don't add information).

snemetz · 2020-10-19T17:43:49Z

This does not seem to be a registry issue
I can run curl with the same url successfully before and after the failed terraform run.
I've done this on a number of days, always with the same results

Unless this can be figured out, 0.13 is a non-starter at my company

snemetz · 2020-10-19T19:14:42Z

Not sure if this will help. But I removed ~/.terraform and the error changed.
Now it displays

`
Initializing provider plugins...

Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not connect to registry.terraform.io: Failed to request discovery
document: Get "https://registry.terraform.io/.well-known/terraform.json":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
`

And via curl
curl https://registry.terraform.io/.well-known/terraform.json {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}

justincampbell · 2020-10-19T20:14:41Z

@snemetz Thanks for the additional info! We're still not able to find any indication of failing requests at our application or CDN layer. Could you try a more detailed cURL request? I don't have the exact headers Terraform sends off-hand, but something like:

curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

I'll try to recreate the exact headers and options now and update you when I have a better arg set.

Also, where are you running these commands from? Are you on a home or office network, VPN, or if you're running this within a cloud provider, which provider and region?

justincampbell · 2020-10-19T20:53:41Z

I did verify that the cURL above is the request that Terraform is making:

Accept-Encoding: gzip
Connection: keep-alive
Host: registry.terraform.io
User-Agent: Terraform/0.13.4
X-Terraform-Version: 0.13.4

snemetz · 2020-10-20T02:23:29Z

curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

Works but terraform fails right afterward. In same shell

$ terraform13.4 init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

I'm running from home on a Mac.

mildwonkey · 2020-10-20T13:31:26Z

Well this is Officially Weird!

We do update the go version and various dependencies with pretty much every release, and it is always possible that something in there is interacting with your local setup.

Is there anything customized or atypical about your home network? Do you have a proxy/VPN/firewall/custom dns settings? What version mac os are you on?

fawaf · 2020-11-29T01:41:40Z

still occurs on Terraform v0.13.5

0dragosh · 2021-01-12T15:32:33Z

Still having this issue on v0.14.4.

paultyng · 2021-01-12T17:29:06Z

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

0dragosh · 2021-01-12T17:35:19Z

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

Timing coincides, thanks a lot for letting me know!

LBoraz · 2021-01-15T16:10:33Z

i have the issue in 0.14.4 with the kubectl provider. For some reason it tries to download it from hashicorp/kubectl instead of gavinbunney/kubectl

Could not retrieve the list of available versions for provider
hashicorp/kubectl: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/kubectl

pselle · 2021-01-15T17:07:24Z

@LBoraz Thank you for trying to connect to a relevant issue, but I think you've actually run into something else! We also have discussion forums that might help in the future.

What I suspect is happening for you is that you have a module that doesn't have the required_providers block declared. Required providers/provider source is a per-module concept, so if you have a module that doesn't have it declared, Terraform is trying to find it in the default namespace (hashicorp) and failing, thus your error.

carmine73 · 2021-03-12T11:27:35Z

The issue is still present

$ terraform version 
Terraform v0.14.8

In my case only if I use $HOME/.terraform.d/plugins as plugin_cache_dir (as @techdragon suggested)

$ cat .terraformrc 
# plugins path
# NOT working
plugin_cache_dir   = "$HOME/.terraform.d/plugins"
# working
#plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
#plugin_cache_dir   = "$HOME/.terraform.d/dummy"

this is my version.tf used just to test terraform init

$ cat versions.tf 
terraform {
  # providers
  required_providers {
    vcd = {
      source = "vmware/vcd"
      version = "3.2.0"
    }
  }
  # terraform version
  required_version = ">= 0.14"
}

MaxiPalle · 2021-04-19T10:19:34Z

I still have this issue with 0.15.0. Any updates on this?

srinavin7 · 2021-05-08T12:48:33Z

Same here. I have this issue with 0.15.0

Lucisano · 2021-05-10T13:59:24Z

Also having this issue

afry-siam · 2021-05-11T17:20:04Z

I'm still getting this issue too, MacOS 11.2.3, Terraform verison 0.15.3

Downgraded to Terraform 0.12.31, and now I can run terraform init successfully!

This problem completely borks a fresh install of Terraform. I couldn't even get copy-pasted example code from the official docs running, because I could never get past terraform init.

Seems like a cool tool, I hope this gets resolved soon! For the meantime I'm downgrading to the earlier version.

For those also seeking to downgrade Terraform Switcher has worked great.

jonstelly · 2021-05-24T22:40:39Z

It seems like this issue may have multiple underlying causes. For me, it seems to be a DNS issue.

I originally shared a log here about seeing this error in WSL2, but like others if I downgraded to terraform 0.13 things worked fine.

@apparentlymart replied there about differences between 0.13 and 0.14 but my issue went away for a couple days so I had nothing to go on. Since then, I've been 100% unable to run terraform init unless I do the following:

If I modify my /etc/resolv.conf to switch my DNS server to 8.8.8.8 (google dns) all terraform commands work as expected. If I switch it back to my standard/default nameserver that WSL assigns, I am completely unable to run any terraform commands that hit the network.

If I turn on TF_LOG I see things like:

2021-05-24T17:30:32.735-0500 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/random/versions
2021-05-24T17:30:42.737-0500 [ERROR] GET https://registry.terraform.io/v1/providers/hashicorp/random/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/random/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

While I can have a second terminal window open looping on curl https://registry.terraform.io/v1/providers/hashicorp/kubernetes/versions and I will get 0 failures.

@apparentlymart mentioned some possible TLS changes, but if this were just a TLS or SSL issue, I don't believe changing nameservers would reliably resolve the issue for me. Below are my results of dig ..., one to my default nameserver, and another to google's dns server.

Editorial: I love terraform but this really needs to get resolved. Look at the number of issues if you search for the client timeout error message. I know those issues go back a few years and terraform has a lot of users but most of those issues have a good number of comments and other people commenting that they're having the same issue. There has to be some fundamental networking issue in terraform or the http client library it's using, otherwise simultaneous curl requests would be failing just like terraform.

I'm going to find some time to try and compile and run terraform from source, if one of the developers wants to add a mess of logging output or try anything to resolve this, send me a branch name and I'll happily test it out and help as much as I can.

dig registry.terraform.io '@172.25.80.1'

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io @172.25.80.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65509
;; flags: qr rd ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  0       IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.66.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.2.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.130.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.194.49
a.gtld-servers.net.     0       IN      A       192.5.6.30
b.gtld-servers.net.     0       IN      A       192.33.14.30
c.gtld-servers.net.     0       IN      A       192.26.92.30
d.gtld-servers.net.     0       IN      A       192.31.80.30
e.gtld-servers.net.     0       IN      A       192.12.94.30
f.gtld-servers.net.     0       IN      A       192.35.51.30
g.gtld-servers.net.     0       IN      A       192.42.93.30
h.gtld-servers.net.     0       IN      A       192.54.112.30

;; Query time: 40 msec
;; SERVER: 172.25.80.1#53(172.25.80.1)
;; WHEN: Mon May 24 17:16:08 CDT 2021
;; MSG SIZE  rcvd: 484

dig registry.terraform.io '@8.8.8.8'

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15680
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  119     IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.2.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.66.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.130.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.194.49

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon May 24 17:16:02 CDT 2021
;; MSG SIZE  rcvd: 165

brookemarshassoc · 2021-06-04T17:22:36Z

I have the same issue above. Resolved it by updating my DNS settings.

xcollantes · 2021-06-12T06:21:31Z

I turned on my VPN and this resolved my issue. From what I read in other posts there is an issue with the DNS resolution with this version of Terraform.

Before VPN:

I could access curl https://registry.terraform.io/v1/providers/hashicorp/google/versions

Using the terraform init command returned the error:

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/google: could not query provider registry for
│ registry.terraform.io/hashicorp/google: the request failed after 2 attempts, please try again later: Get
│ "https://registry.terraform.io/v1/providers/hashicorp/google/versions": context deadline exceeded

After turning on VPN:

terraform init works as intended

kindacoder · 2021-06-24T06:38:02Z

I have the same issue above. Resolved it by updating my DNS settings.

what change you did ?

kindacoder · 2021-06-24T06:38:49Z

#26532 (comment)
what change you did @brookemarshassoc

jbardin · 2021-06-24T16:40:13Z

Hello,

Starting with Terraform v1.0.1 the MacOS binary is natively compiled, allowing Terraform to use the host resolver. This should avoid most issues with DNS configuration, many of which show up as timeouts like shown here.

Since this will likely resolve most of the issues represented by this thread I'm going to close it out. If anyone still encounters similar timeouts from the Terraform CLI on MacOS with v1.0.1 or newer, please file a new issue with the necessary information to reproduce it.

Thanks!

jonstelly · 2021-06-24T17:47:18Z

I'm still seeing this on WSL2 under windows. I'll get clean repro steps and create a new issue today or tomorrow.

zzxwill · 2021-06-29T09:18:29Z

I am still getting this issue with Terraform 1.0.1 on an alpine docker image.

bash-5.1# ./terraform init
2021-06-29T09:15:06.197Z [DEBUG] Adding temp file log sink: /tmp/terraform-log106468856
2021-06-29T09:15:06.197Z [INFO]  Terraform version: 1.0.1
2021-06-29T09:15:06.197Z [INFO]  Go runtime version: go1.16.4
2021-06-29T09:15:06.197Z [INFO]  CLI args: []string{"/data/terraform", "init"}
2021-06-29T09:15:06.197Z [TRACE] Stdout is a terminal of width 175
2021-06-29T09:15:06.197Z [TRACE] Stderr is a terminal of width 175
2021-06-29T09:15:06.197Z [TRACE] Stdin is a terminal
2021-06-29T09:15:06.197Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2021-06-29T09:15:06.197Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-06-29T09:15:06.197Z [DEBUG] checking for credentials in "/root/.terraform.d/plugins"
2021-06-29T09:15:06.197Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-06-29T09:15:06.197Z [DEBUG] will search for provider plugins in /root/.terraform.d/plugins
2021-06-29T09:15:06.198Z [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/alicloud v1.120.0 for linux_amd64 at /root/.terraform.d/plugins/registry.terraform.io/hashicorp/alicloud/1.120.0/linux_amd64
2021-06-29T09:15:06.198Z [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/aws v3.36.0 for linux_amd64 at /root/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/3.36.0/linux_amd64
2021-06-29T09:15:06.198Z [DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
2021-06-29T09:15:06.198Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2021-06-29T09:15:06.198Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2021-06-29T09:15:06.199Z [INFO]  CLI command args: []string{"init"}
Initializing modules...
2021-06-29T09:15:06.203Z [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2021-06-29T09:15:06.204Z [DEBUG] Module installer: begin vpc
2021-06-29T09:15:06.204Z [TRACE] ModuleInstaller: vpc is not yet installed
2021-06-29T09:15:06.204Z [TRACE] ModuleInstaller: cleaning directory .terraform/modules/vpc prior to install of vpc
2021-06-29T09:15:06.204Z [TRACE] ModuleInstaller: vpc is a registry module at terraform-google-modules/network/google
2021-06-29T09:15:06.204Z [DEBUG] vpc listing available versions of terraform-google-modules/network/google at registry.terraform.io
2021-06-29T09:15:06.204Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2021-06-29T09:15:06.204Z [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2021-06-29T09:15:09.201Z [ERROR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=2884c382-efd8-0c50-23d5-16d0383a2b06&version=1.0.1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021-06-29T09:15:17.204Z [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json
╷
│ Error: Error accessing remote module registry
│
│ Failed to retrieve available versions for module "vpc" (main.tf:1) from registry.terraform.io: Failed to request discovery document: Get
│ "https://registry.terraform.io/.well-known/terraform.json": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
╵

╷
│ Error: Error accessing remote module registry
│
│ Failed to retrieve available versions for module "vpc" (main.tf:1) from registry.terraform.io: Failed to request discovery document: Get
│ "https://registry.terraform.io/.well-known/terraform.json": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
╵



bash-5.1# curl https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=6935a4be-5f95-e966-58da-a1fe8d135790&version=0.14.10
[1] 505
[2] 506
[3] 507
[2]-  Done                    os=linux
bash-5.1# {"product":"terraform","current_version":"1.0.1","current_release":1624550981,"current_download_url":"https://releases.hashicorp.com/terraform/1.0.1/","current_changelog_url":"https://github.com/hashicorp/terraform/blob/v1.0.1/CHANGELOG.md","project_website":"https://www.terraform.io","alerts":[]}
[1]-  Done                    curl https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64
[3]+  Done                    signature=6935a4be-5f95-e966-58da-a1fe8d135790

kindacoder · 2021-06-29T18:53:44Z

The issue is with dns on your mac. Open your network preferences and go to DNS servers listing tab and delete all the ip addresses there. the issue will be fixed. Took me 2 days to figure it out.

raoim · 2021-07-27T21:18:54Z

Yeah setting DNS to 8.8.8.8 resolved my issues
https://www.support.com/how-to/how-to-change-dns-settings-on-a-mac-10189

github-actions · 2021-08-27T02:06:28Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

snemetz added bug new new issue not yet triaged labels Oct 9, 2020

pselle added the registry label Oct 14, 2020

pselle added waiting for reproduction unable to reproduce issue without further information and removed new new issue not yet triaged labels Oct 14, 2020

pselle added new new issue not yet triaged and removed registry labels Oct 14, 2020

mildwonkey changed the title ~~Error: Failed to query available provider packages~~ Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers Oct 19, 2020

apparentlymart added v0.13 Issues (primarily bugs) reported against v0.13 releases v0.14 Issues (primarily bugs) reported against v0.14 releases labels Dec 2, 2020

jbardin mentioned this issue Dec 4, 2020

Terraform 0.13, 0.14 performance terrible using binary, good when self built #27143

Closed

sjpb mentioned this issue Feb 11, 2021

Client.Timeout exceeded while awaiting headers (Linux) #27742

Closed

jbardin mentioned this issue Mar 26, 2021

net/http: TLS handshake timeout | PLEASE FIX IT FINALLY #28214

Closed

jbardin mentioned this issue Jun 10, 2021

'terraform init' does not work on Mac OSX Big Sur #28917

Closed

jbardin closed this as completed Jun 24, 2021

github-actions bot locked as resolved and limited conversation to collaborators Aug 27, 2021

Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers #26532

Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers #26532

Comments

snemetz commented Oct 9, 2020 • edited Loading

Terraform Version

Terraform Configuration Files

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional Context

crosbymichael1 commented Oct 13, 2020 • edited by pselle Loading

Error Message:

pselle commented Oct 14, 2020 • edited Loading

crosbymichael1 commented Oct 14, 2020 • edited Loading

pselle commented Oct 14, 2020

crosbymichael1 commented Oct 14, 2020 • edited Loading

snemetz commented Oct 15, 2020

snemetz commented Oct 15, 2020

techdragon commented Oct 16, 2020

techdragon commented Oct 16, 2020 • edited Loading

C-Howard commented Oct 19, 2020

mildwonkey commented Oct 19, 2020

Error: Failed to query available provider packages w/net/http: request canceled while waiting for connection

Terraform not updating provider version

Issues when setting plugin_cache_dir to implicit local mirror directory #26223

mildwonkey commented Oct 19, 2020

C-Howard commented Oct 19, 2020 • edited Loading

mildwonkey commented Oct 19, 2020

C-Howard commented Oct 19, 2020 • edited Loading

mildwonkey commented Oct 19, 2020

snemetz commented Oct 19, 2020 • edited Loading

snemetz commented Oct 19, 2020 • edited Loading

justincampbell commented Oct 19, 2020

justincampbell commented Oct 19, 2020

snemetz commented Oct 20, 2020 • edited Loading

mildwonkey commented Oct 20, 2020 • edited Loading

fawaf commented Nov 29, 2020

0dragosh commented Jan 12, 2021 • edited Loading

paultyng commented Jan 12, 2021

0dragosh commented Jan 12, 2021

LBoraz commented Jan 15, 2021

pselle commented Jan 15, 2021

carmine73 commented Mar 12, 2021 • edited Loading

MaxiPalle commented Apr 19, 2021

srinavin7 commented May 8, 2021

Lucisano commented May 10, 2021

afry-siam commented May 11, 2021 • edited Loading

jonstelly commented May 24, 2021 • edited Loading

brookemarshassoc commented Jun 4, 2021

xcollantes commented Jun 12, 2021

Before VPN:

After turning on VPN:

kindacoder commented Jun 24, 2021

kindacoder commented Jun 24, 2021

jbardin commented Jun 24, 2021

jonstelly commented Jun 24, 2021

zzxwill commented Jun 29, 2021 • edited Loading

kindacoder commented Jun 29, 2021 • edited Loading

raoim commented Jul 27, 2021

github-actions bot commented Aug 27, 2021

snemetz commented Oct 9, 2020 •

edited

Loading

crosbymichael1 commented Oct 13, 2020 •

edited by pselle

Loading

pselle commented Oct 14, 2020 •

edited

Loading

crosbymichael1 commented Oct 14, 2020 •

edited

Loading

crosbymichael1 commented Oct 14, 2020 •

edited

Loading

techdragon commented Oct 16, 2020 •

edited

Loading

C-Howard commented Oct 19, 2020 •

edited

Loading

C-Howard commented Oct 19, 2020 •

edited

Loading

snemetz commented Oct 19, 2020 •

edited

Loading

snemetz commented Oct 19, 2020 •

edited

Loading

snemetz commented Oct 20, 2020 •

edited

Loading

mildwonkey commented Oct 20, 2020 •

edited

Loading

0dragosh commented Jan 12, 2021 •

edited

Loading

carmine73 commented Mar 12, 2021 •

edited

Loading

afry-siam commented May 11, 2021 •

edited

Loading

jonstelly commented May 24, 2021 •

edited

Loading

zzxwill commented Jun 29, 2021 •

edited

Loading

kindacoder commented Jun 29, 2021 •

edited

Loading