Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error loading modules: host registry.terraform.io #16751

Closed
thunderbird86 opened this issue Nov 23, 2017 · 23 comments
Closed

Error loading modules: host registry.terraform.io #16751

thunderbird86 opened this issue Nov 23, 2017 · 23 comments

Comments

@thunderbird86
Copy link

Terraform Version

Terraform v0.11.0

Terraform Configuration Files

module "vpc" {
    source = "terraform-aws-modules/vpc/aws"
}
module "vpc" {
    source = "terraform-aws-modules/vpc/aws"
    version = "1.4.0"
}

Debug Output

2017/11/23 12:01:17 [TRACE] module source: "terraform-aws-modules/vpc/aws"
2017/11/23 12:01:17 [TRACE] "terraform-aws-modules/vpc/aws" is a registry module
2017/11/23 12:01:17 [INFO] no matching version for "terraform-aws-modules/vpc/aws"<1.4.0>, no versions found
2017/11/23 12:01:17 [DEBUG] matched "terraform-aws-modules/vpc/aws" version  for 1.4.0
2017/11/23 12:02:24 [TRACE] module source: "terraform-aws-modules/vpc/aws"
2017/11/23 12:02:24 [TRACE] "terraform-aws-modules/vpc/aws" is a registry module
2017/11/23 12:02:24 [INFO] no matching version for "terraform-aws-modules/vpc/aws"<>, no versions found
2017/11/23 12:02:24 [DEBUG] matched "terraform-aws-modules/vpc/aws" version  for 

Expected Behavior

Download modules from registry

Actual Behavior

Can't download modules from registry

Steps to Reproduce

terraform init 
Upgrading modules...
- module.vpc
Error downloading modules: Error loading modules: host registry.terraform.io does not provide Terraform modules
@ebarault
Copy link

ebarault commented Nov 23, 2017

experienced the same issue just now. It was working yesterday.
using github directly works:

  source = "[email protected]:terraform-aws-modules/terraform-aws-vpc.git"

@arunkspkm
Copy link

@thunderbird86 It's working fine for me after I rewrite module to below format.
module "vpc" {
source = "./terraform-aws-modules/vpc/aws"
}

@jbardin
Copy link
Member

jbardin commented Nov 28, 2017

Hi @thunderbird86,

This is covered in the upgrade guide. Relative paths must start with ./ or ../ to differentiate them from named registry modules.

@jbardin jbardin closed this as completed Nov 28, 2017
@thunderbird86
Copy link
Author

thunderbird86 commented Nov 28, 2017

Hi @jbardin

Did you read issue?

Expected Behavior

Download modules from registry

It is registry module. this

And in trace

2017/11/23 12:01:17 [TRACE] "terraform-aws-modules/vpc/aws" is a registry module

@jbardin
Copy link
Member

jbardin commented Nov 28, 2017

Hi @thunderbird86,

Sorry, the last comment made me think that this was the issue with ambiguous names. That log message only indicates the name conforms to a registry module, not that the source has been located.

Is this the extent of your configuration, and is it still repeatable? Using that in a terraform config works fine here.

@jbardin jbardin reopened this Nov 28, 2017
@thunderbird86
Copy link
Author

Hi @jbardin

It's have very strange behavior. Right now this module works fine.

But i try to init kubernetes module, and have the same issue on Ubuntu 16.04, and at Mac all looks fine

module "kubernetes" {
    source = "coreos/kubernetes/aws"

    tectonic_admin_email = "[email protected]"
    tectonic_admin_password = "1111111"
    tectonic_aws_external_vpc_id = ""
    tectonic_aws_extra_tags = ""
    tectonic_aws_master_custom_subnets = ""
    tectonic_aws_master_ec2_type = "t2.medium"
    tectonic_aws_region = ""
    tectonic_aws_ssh_key = ""
    tectonic_aws_vpc_cidr_block = "10.200.0.0/16"
    tectonic_aws_worker_ec2_type = "t2.medium"
}
Error downloading modules: Error loading modules: host registry.terraform.io does not provide Terraform modules

@ebarault
Copy link

ebarault commented Nov 28, 2017

using
source = "./terraform-aws-modules/vpc/aws"
instead of
source = "terraform-aws-modules/vpc/aws"
to make it work in 0.11 with official registry seems a very obscure convention to me...

@jbardin
Copy link
Member

jbardin commented Nov 28, 2017

@ebarault, that is the case only for relative file paths, which was the case with a earlier comment, but the original issue posted is for a registry module.

@ebarault
Copy link

ebarault commented Nov 28, 2017

@jbardin : ok, so we agree this

module "vpc" {
    source = "terraform-aws-modules/vpc/aws"
}

should work?

@jbardin
Copy link
Member

jbardin commented Nov 28, 2017

@ebarault: yes, that's the correct source for a registry module, which works in most cases.

@thunderbird86: Are you possibly running behind a proxy of some sort? Can you provide the full log up to the does not provide Terraform modules line?

@jbardin
Copy link
Member

jbardin commented Nov 28, 2017

For reference, I'm not able to replicate this using various http/https proxy configuration either, though I of course can't account for broken proxies.

@thunderbird86: do you possibly have any host blocks in a global .terraformrc file?

@apparentlymart
Copy link
Contributor

apparentlymart commented Nov 29, 2017

It may help to give some additional context about what Terraform is trying to do behind the scenes here, that might be a clue as to how it might be interacting poorly with a local proxy or other such intermediary on your networks.

This "does not provide Terraform modules" error is a result of failing to complete the service discovery protocol. You can see the full details of that on that page, but the crux of it is that Terraform wants to fetch https://registry.terraform.io/.well-known/terraform.json and get back a JSON document that gives the base URL for the modules.v1 service.

If Terraform gets back anything other than a JSON document at this URL, discovery is considered to have failed. Terraform will follow at least one redirect, but still expects the final URL to return a JSON document.

One potential cause of this error, for example, would be if you're on a wireless network which uses a "captive portal" to force a login or agree to terms and conditions. This would cause that discovery request to fail (to be blocked outright or to fail its TLS certificate check) and thus cause this error.


James mentioned the possibility of host blocks in the .terraformrc file, which as well as being a possible cause of strange behavior could also be a reasonable workaround for this problem, if it's only the discovery that's failing. Putting the following in the CLI config should circumvent the discovery check and go straight to the module API:

# NOTE WELL: The URL declared in here is subject to change at any time, which will
# cause module installation to fail. It is not recommended to leave the following permanently
# in the CLI config, since real network discovery is required to ensure that Terraform will
# automatically adopt any future changes to the module API URL.
host "registry.terraform.io" {
  services = {
    "modules.v1" = "https://registry.terraform.io/v1/modules/"
  }
}

It could be informative to try this and see if that makes it work, or if that just leads to a further error accessing the registry API. If the latter, that suggests a general problem with the local network connection, whereas the former could be indicative of a bug in the discovery client.

@thunderbird86
Copy link
Author

@jbardin have no proxy.

Here full trace log

2017/11/29 10:30:46 [INFO] Terraform version: 0.11.0  ec9d4f1d0f90e8ec5148f94b6d634eb542a4f0ce+CHANGES
2017/11/29 10:30:46 [INFO] Go runtime version: go1.9
2017/11/29 10:30:46 [INFO] CLI args: []string{"/opt/bin/terraform/terraform", "init"}
2017/11/29 10:30:46 [DEBUG] Attempting to open CLI config file: /home/art/.terraformrc
2017/11/29 10:30:46 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2017/11/29 10:30:46 [INFO] CLI command args: []string{"init"}
2017/11/29 10:30:46 [DEBUG] command: loading backend config file: /home/art/tmp/tf
2017/11/29 10:30:46 [TRACE] module source: "terraform-aws-modules/vpc/aws"
2017/11/29 10:30:46 [TRACE] "terraform-aws-modules/vpc/aws" is a registry module
2017/11/29 10:30:46 [INFO] no matching version for "terraform-aws-modules/vpc/aws"<>, no versions found
2017/11/29 10:30:46 [DEBUG] matched "terraform-aws-modules/vpc/aws" version  for 
2017/11/29 10:30:46 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2017/11/29 10:30:49 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=10f42454-d52b-baac-fd29-053d7390c731&version=0.11.0: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2017/11/29 10:30:50 [WARNING] Failed to request discovery document: Get https://registry.terraform.io/.well-known/terraform.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2017/11/29 10:30:50 [DEBUG] plugin: waiting for all plugin processes to complete...

And curl works
curl https://registry.terraform.io/v1/modules/terraform-aws-modules/vpc/aws/1.4.0 | jq .

@thunderbird86
Copy link
Author

@apparentlymart yes it's helps.
I create .terrafromrc with content

host "registry.terraform.io" {
  services = {
    "modules.v1" = "https://registry.terraform.io/v1/modules/"
  }
}

@jbardin
Copy link
Member

jbardin commented Nov 29, 2017

Interesting, these are http timeouts causing the failures. While the other instances I've seen of this are TLS handshake timeouts, it might be related to #16448

@apparentlymart
Copy link
Contributor

Some other interesting things I notice here:

  • The request to the Checkpoint API seems to be failing in a similar way. These are two separate applications, so it seems unlikely that they would have a server-side application-level bug in common here.
  • Both registry.terraform.io and checkpoint-api.hashicorp.com are hosted on Heroku at the time of writing (subject to change in future, of course) and are thus likely to be taking similar routes over the Internet.
  • Regarding the potential link with Intermittent net/http: TLS handshake timeout error when downloading providers #16448: releases.hashicorp.com is, on the other hand, fronted by Fastly at the time of writing, and is thus likely to be taking pretty different routes and is probably interacting with different software at the remote end that may behave slightly differently in the event of packet loss or delay.
  • The difference for Intermittent net/http: TLS handshake timeout error when downloading providers #16448 could also be explained by different handling of timeouts on Terraform's end: both the checkpoint client and the discovery client set timeouts at the HTTP client layer to avoid "hangs" when the remote service is down or unreachable. The plugin installer, on the other hand, seems to set no explicit HTTP timeout and is thus bound instead by the timeout on the TCP socket itself, causing the error to surface at the crypto/tls layer instead of the net/http layer as we saw here.
  • It's interesting that bypassing the discovery protocol using the .terraformrc override caused this to work for @thunderbird86, even though the module API is being served from registry.terraform.io too. Notable though that module installation uses the same mechanism as plugin installation, and so probably also lacks any explicit timeout at the HTTP client layer.

We currently have the discovery timeout set to four seconds and the checkpoint timeout set to three seconds, while the TCP socket timeouts are at their defaults and thus significantly longer. It may just be that four seconds is too aggressive a timeout for users in some locations and we should consider increasing it.

@jbardin
Copy link
Member

jbardin commented Nov 29, 2017

The TLS handshake has a separate timeout at 10 seconds, so not significantly longer.

@jbardin
Copy link
Member

jbardin commented Nov 29, 2017

These are the same error. From the client's perspective, it is waiting for headers even if it's stalled at a lower layer. I verified that you get Client.Timeout exceeded while awaiting headers error if the client timeout is shorter, and TLS handshake timeout if the TLSHandshakeTimeout is shorter.

@kinghuang
Copy link

kinghuang commented Sep 26, 2018

I'm having this problem all of a sudden, coincidentally with the terraform-aws-modules/vpc/aws module.

▸ terraform init
2018/09/26 17:32:32 [INFO] Terraform version: 0.11.8  
2018/09/26 17:32:32 [INFO] Go runtime version: go1.10.3
2018/09/26 17:32:32 [INFO] CLI args: []string{"/usr/local/Cellar/terraform/0.11.8/bin/terraform", "init"}
2018/09/26 17:32:32 [DEBUG] Attempting to open CLI config file: /Users/king.huang/.terraformrc
2018/09/26 17:32:32 Loading CLI configuration from /Users/king.huang/.terraformrc
2018/09/26 17:32:32 [INFO] CLI command args: []string{"init"}
2018/09/26 17:32:32 [DEBUG] command: loading backend config file: /Users/king.huang/Repositories/devops/scratch/vpc-testing
Initializing modules...
2018/09/26 17:32:32 [TRACE] module source: "../../terraform/terraform-aws-vpc/"
- module.vpc
2018/09/26 17:32:32 [TRACE] module source: "terraform-aws-modules/vpc/aws"
2018/09/26 17:32:32 [TRACE] "terraform-aws-modules/vpc/aws" is a registry module
- module.vpc.vpc
2018/09/26 17:32:32 [DEBUG] found local version "1.37.0" for module terraform-aws-modules/vpc/aws
2018/09/26 17:32:32 [INFO] no matching version for "terraform-aws-modules/vpc/aws"<1.44.0>, no version found matching constraint "1.44.0"
2018/09/26 17:32:32 [DEBUG] matched "terraform-aws-modules/vpc/aws" version  for 1.44.0
2018/09/26 17:32:32 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2018/09/26 17:32:35 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=darwin&signature=8cf7d926-59e8-cbb2-87a7-fedfda81c59a&version=0.11.8: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018/09/26 17:32:42 [WARN] Failed to request discovery document: Get https://registry.terraform.io/.well-known/terraform.json: net/http: TLS handshake timeout
2018/09/26 17:32:42 [DEBUG] plugin: waiting for all plugin processes to complete...
Error downloading modules: Error loading modules: host registry.terraform.io does not provide Terraform modules
▸ time curl https://registry.terraform.io/.well-known/terraform.json
{"modules.v1":"/v1/modules/"}
curl https://registry.terraform.io/.well-known/terraform.json  0.03s user 0.01s system 15% cpu 0.203 total
▸ time curl https://checkpoint-api.hashicorp.com/v1/check/terraform\?arch\=amd64\&os\=darwin\&signature\=8cf7d926-59e8-cbb2-87a7-fedfda81c59a\&version\=0.11.8
{"product":"terraform","current_version":"0.11.8","current_release":1534374840,"current_download_url":"https://releases.hashicorp.com/terraform/0.11.8/","current_changelog_url":"https://github.com/hashicorp/terraform/blob/v0.11.8/CHANGELOG.md","project_website":"https://www.terraform.io","alerts":[]}
curl   0.03s user 0.01s system 9% cpu 0.352 total

The requests for the service discovery and checkpoint URLs always succeed immediately with curl, but terraform has problems with them. I'm not going through any proxies, and there's no captive portals on my WiFi network. I'm at a loss on how to get past this problem.

Here's the requests with headers.

Click to expand

http https://registry.terraform.io/.well-known/terraform.json
HTTP/1.1 200 OK
Accept-Ranges: bytes
Accept-Ranges: bytes
Accept-Ranges: bytes
Accept-Ranges: bytes
Accept-Ranges: bytes
Age: 0
Age: 0
Age: 0
Cache-Control: public, stale-if-error=31536000
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 49
Content-Type: application/json
Date: Wed, 26 Sep 2018 23:43:23 GMT
Last-Modified: Thu, 02 Aug 2018 15:11:06 GMT
Server: Cowboy
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
Via: 1.1 vegur
Via: 1.1 varnish
Via: 1.1 varnish
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Served-By: cache-iad2140-IAD, cache-sea1040-SEA

{
    "modules.v1": "/v1/modules/"
}

http https://checkpoint-api.hashicorp.com/v1/check/terraform\?arch\=amd64\&os\=darwin\&signature\=8cf7d926-59e8-cbb2-87a7-fedfda81c59a\&version\=0.11.8
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Accept, Cache-Control, X-Requested-With, Content-Type
Access-Control-Allow-Methods: GET, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 86400
Connection: keep-alive
Content-Length: 301
Content-Type: application/json
Date: Wed, 26 Sep 2018 23:43:50 GMT
Server: Cowboy
Via: 1.1 vegur

{
    "alerts": [],
    "current_changelog_url": "https://github.com/hashicorp/terraform/blob/v0.11.8/CHANGELOG.md",
    "current_download_url": "https://releases.hashicorp.com/terraform/0.11.8/",
    "current_release": 1534374840,
    "current_version": "0.11.8",
    "product": "terraform",
    "project_website": "https://www.terraform.io"
}

@kinghuang
Copy link

kinghuang commented Sep 28, 2018

I don't think there's actually any network connectivity issues, at least for my case. Perhaps terraform on my Mac is suddenly unable to validate the TLS certificate? Or, there's a library or dependency somewhere that's changed?

I created a simple config using the AWS VPC module to test with, and ran Terraform directly on my Mac and in a Docker container on my Mac. I'm on macOS High Sierra, with Terraform v0.11.8 installed via homebrew.

module "test" {
  source = "terraform-aws-modules/vpc/aws"
  version = "1.44.0"
}

Running terraform init directly on my Mac, I get the Client.Timeout exceeded while awaiting headers error.

Click to expand

$ terraform init
Initializing modules...
- module.test
Error downloading modules: Error loading modules: Get https://registry.terraform.io/v1/modules/terraform-aws-modules/vpc/aws/versions: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)


If I use the hashicorp/terraform:light image and run Terraform in Docker against the same directory, it works.

Click to expand

$ docker container run -it --rm -v $PWD:/tf --workdir /tf hashicorp/terraform:light init
Initializing modules...
- module.test
  Found version 1.44.0 of terraform-aws-modules/vpc/aws on registry.terraform.io
  Getting source "terraform-aws-modules/vpc/aws"

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.38.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.38"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.


I've tried several different network locations, and consistently cannot init with my natively installed copy of Terraform anymore, but can with Terraform in the container. This feels more like something that Terraform depends on has changed. Though, I don't know what.

@Constantin07
Copy link

Getting the same issue on 0.12-dev branch but works OK on 0.11.10.

@m-dekkers
Copy link

I am also seeing this issue, with terraform 0.11.14, on a Mac. No proxies, captive portals, or anything similar. fetching the url from curl or the browser is instant, but terraform doesn't like it

@ghost
Copy link

ghost commented Oct 2, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Oct 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants