Incident summary for Terraform Registry module download failures (affects TF 0.12.0 to 0.12.18)

[justincampbell]

Terraform 0.12 contained a bug with go-getter 1.3.0, where HEAD requests which did not return a Content-Length header in the response were not handled properly. This was fixed in go-getter 1.4.1 and Terraform 0.12.19. The fix is detailed in hashicorp/go-getter#227.

The Terraform Registry uses GitHub for both user authentication and module publishing/hosting. The go-getter issue was discovered because in mid-December 2019, GitHub began omitting the Content-Length header from HEAD requests for some percentage of their traffic. On January 9th, 2020, it became apparent that all requests now exhibit this behavior, and that all Terraform Registry module downloads for these Terraform versions were affected.

Today (January 10th, 2020), we deployed a change to the Terraform Registry which detects requests from the affected Terraform versions and returns a Git go-getter URL (as opposed to HTTPS), which should allow these versions to successfully install modules from the Terraform Registry.

Although we believe this workaround will solve the issue for most users, we recommend that all Terraform 0.12 users upgrade to the latest version (TF 0.12.19 or later). Thanks!

[kxavier-ims]

Hi,

What about terraform 0.11.x? We are experiencing the same problems downloading the modules and we can't upgrade to 0.12 in the short term.
The error we are getting is "Error downloading modules: Error loading modules: subdir "*" not found"

[justincampbell]

@kxavier-ims Our understanding was that Terraform 0.11.x was not affected, because it contains a version of go-getter prior to the introduction of this bug. Could you open an issue with the specific version you're using, and the output of TF_LOG=1 terraform init? Thanks!

[wershlak]

This is not fixed for us with 0.12.12. I don't see any change to the behavior or the url from yesterday. Are you sure the change is deployed?

We're currently migrating from tf 11 to 12 and can't upgrade to 0.12.19 until that is complete.

[gkowalski-google]

This is not fixed for us with 0.12.12. I don't see any change to the behavior or the url from yesterday. Are you sure the change is deployed?

We're currently migrating from tf 11 to 12 and can't upgrade to 0.12.19 until that is complete.

We are seeing the same issue. When I am testing locally on a mac with 0.12.6, I have not run into this issue at all. However, our test pipeline running Google Cloud Build with 0.12.6 reproduces the error every time. Thanks for trying to address this!

[justincampbell]

@wershlak @gkowalski-google We've identified that we may not be taking the Terraform version into account when caching this response. We're looking into it now.

[justincampbell]

@wershlak @gkowalski-google Could you try now and let me know if it works? Thanks!

[gkowalski-google]

@justincampbell Re-ran one test and we did not get the error! I am going to run some more tests but so far it's looking promising. Thank you so much!

[wershlak]

Looks like downloads are working now. Thanks for reopening this and getting it fixed @justincampbell

[kxavier-ims]

@kxavier-ims Our understanding was that Terraform 0.11.x was not affected, because it contains a version of go-getter prior to the introduction of this bug. Could you open an issue with the specific version you're using, and the output of TF_LOG=1 terraform init? Thanks!

Hi @justincampbell . We were using a patched version of 0.11.x. Now it is working. Thanks.

[hunt3ri]

Hi @justincampbell I'm seeing the "Could not download module" error only in 0.12.19 and 0.12.20. My registry module downloads successfully in 0.12.18. Module code set as follows:

module "vpc" {
    source  = "terraform-aws-modules/vpc/aws"
    version = "~> 2.0"
    ...
}

[TomHowarth]

I am seeing a similar error with 0.12.21

I am downloading my modules from my github repository. and remote state is active on Terraform Cloud.

since moving to this module download has moved from .\terrafrom\modules\services\modulename the expected location to .\terraform\modules\modulename\modules\servcies\modulename.

obviously when running terraform init initialization fails. why is this behaviour happening?

Could not download module "Webserver-cluster" (main.tf:41) source code from
"github.com/REDACTED/REDACTED/modules/servces/webserver-cluster": subdir
"modules\servces\webserver-cluster" not found

Error: Failed to download module

[justincampbell]

Hey @TomHowarth! I don't believe this issue is still affecting Terraform releases since 0.12.19. But I'll check our logs to see if there's a correlation of failures with TF version.

In your description, the word services is spelled servces in the output below. Is it possible there's a typo there that's preventing this init from succeeding?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.