0.10.0-beta1 provider plugins download take long time

[iceycake]

Terraform Version

0.10.0-beta1

Steps to Reproduce

$GOPATH/src/github.com/hashicorp/terraform/bin/terraform init \
-backend-config=bucket=our-s3-bucket \
-backend-config=key=PRDXYZ/nonprod/andyc1/terraformer-test.tfstate \
-backend-config=region=us-east-1 \
-backend-config=profile=ops \
-backend-config=dynamodb_table=terraformer_lock_table \
-lock-timeout=10m

It takes a long time to download just template and aws plugins. The template plugin just 16M and it took > 15mins and aws plugins is still downloading after 30m.

Is this behavior normal?

[iceycake]

Finally got an error:

Error installing provider "aws": read tcp 10.45.240.210:64934->151.101.25.183:443: read: connection reset by peer.

[apparentlymart]

Hi @iceycake! Sorry this is taking so long...

Indeed this seems like strange behavior. Could you try downloading the relevant file for your OS and architecture directly from the repository to see if it's any faster from outside of Terraform's auto-installer?

[iceycake]

@apparentlymart The repository download speed is very slow. 4.5MB needs 5 mins?

[apparentlymart]

@iceycake honestly I can't really explain this. Your Github profile suggests you're in L.A., which would put you close to one of the Fastly CDN distribution points and likely pulling from the same one as several of our staff who are not seeing this problem. Unfortunately I'm not sure what to suggest here 😖 ... is there anything unusual about your Internet connection that might give a clue as to why your experience would be different from others?

[iceycake]

All the internet connections in our office is normal. Anyway, seems nothing much can do about it. It will be great if we can pre-fetch the plugins and hosted internally. I know terraform 0.10 supports --plugin-dir but what about --plugin-url so we can pre-fetch the providers and hosted internally to avoid internet dependencies?

[apparentlymart]

Hi @iceycake,

At the moment we've intentionally not allowed that since the auto-install is depending on some details of how the releases server generates indexes and we want to be able to change that in future releases as we improve things to support third-party plugin downloads, etc. This likely will become possible later, but we need to actually specify the requirements for a third-party-hosted repository before we can do it.

In the mean time, as you saw it's possible to download the providers yourself and make them available to Terraform, though I understand that's less convenient. Along with -plugin-dir it's possible to put the downloaded plugin binaries in the same directory as the terraform binary and then terraform init will find and use them without any additional arguments, which may be more convenient.

[iceycake]

Is there an easy way to download all the providers as a bundle now?

[apparentlymart]

We expect that most users don't need all providers, since it's very rare to have a configuration that uses all of them. However, we did recently add a tool for automatically merging together a core terraform zip file with zero or more providers specified in a configuration file, which may make things easier for you. This tool is primarily aimed at situations where Terraform is running on a machine that can't access releases.hashicorp.com at all, but maybe helps with slow access to that site too.

They will of course still be downloading from the same place as Terraform itself downloads them (it uses the same installation code, in fact) but at least at the end you will have a zip file that you can extract to get everything installed at once.

[iceycake]

Sounds good. Let me try that tool.

[iceycake]

The terraform-bundle tool comes just in time! It serves well for our use case. I think we can close this issue now.

[ghost]

Why is this issue closed? It certainly isn't solved. Working around a network misconfiguration issue is a royal pain in the neck. The network misconfiguration should be corrected. The problem exists between roadrunner and lastly, so is likely a bad BGP advertisement coming from Lastly. It isn't going to fix itself without someone actually making demands of Lastly and/or roadrunner. I've already created a ticket at roadrunner, but the problem is likely at Lastly's end, or will at least require Lastly to assist with correction.

Here's an example traceroute - no matter how many times I run it, it always hangs at that same hop:

$ traceroute releases.hashicorp.com
traceroute to dualstack.s.shared.global.fastly.net (151.101.25.183), 64 hops max, 52 byte packets
 1  linksys26757 (192.168.1.1)  3.493 ms  2.109 ms  5.401 ms
 2  192.168.0.1 (192.168.0.1)  10.805 ms  14.139 ms  13.702 ms
 3  142.254.236.145 (142.254.236.145)  23.638 ms  200.262 ms  28.151 ms
 4  agg60.lsapcawv01h.socal.rr.com (24.30.169.105)  34.880 ms  20.768 ms  31.983 ms
 5  agg20.lsaicaev01r.socal.rr.com (72.129.19.8)  27.503 ms  25.916 ms  27.596 ms
 6  agg26.lsancarc01r.socal.rr.com (72.129.17.0)  27.991 ms  40.311 ms  39.269 ms
 7  bu-ether26.lsancarc0yw-bcr00.tbone.rr.com (66.109.3.230)  31.003 ms  34.239 ms  34.392 ms
 8  bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  35.103 ms
    bu-ether11.tustca4200w-bcr00.tbone.rr.com (66.109.6.5)  36.238 ms
    bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  34.622 ms
 9  0.ae2.pr1.lax10.tbone.rr.com (107.14.19.54)  32.775 ms
    0.ae3.pr1.lax10.tbone.rr.com (107.14.19.56)  31.514 ms
    0.ae2.pr1.lax10.tbone.rr.com (107.14.19.54)  27.672 ms
10  * * *
11  * * *
12  * * *

[ghost]

Incidentally, since this problem appears to exist between a number of networks and Lastly, it is HIGHLY likely that the problem is a bad BGP or router config within Lastly's infrastructure unless all of the other people suffering this problem around the world are all on networks that peer with roadrunner to get to whatever address Lastly is advertising. That's possible, but highly improbable.

Note - CDNs that can't get their own networking correct are likely CDNs one shouldn't be relying on for mission critical infrastructure tooling. There's a reason Akamai is 3x the size of the next biggest CDN.

If the canned response to customers experiencing icredibly slow CDN access is "you have crap connectivity to our CDN" then surely that ought to be reason to change your CDN? After all, isn't the whole point of a CDN to put content CLOSE to users at the edge of the network, rather than to artificially increase the distance between them and the origin? If your CDN is the CAUSE of high latency, get rid of the damn CDN.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.