Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory #27155

joncolby · 2020-12-05T12:46:43Z

This new issue re-raises an issue that I believe was incorrectly closed. Issue #27099 was closed with the reason "duplicate issue of #27092", however the issue used as a reason to close #27099 is not the same use case.

Problem: running terraform init multiple times should re-use previously downloaded provider versions. It seems like terraform providers are always downloaded, regardless of any additional settings.

Due to this issue, the added time to re-download providers with every run causes terraform workflows to take longer

Terraform Version

Terraform v0.14.0

Expected Behavior

Running terraform init 2 or more times, with no provider version changes

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file

Terraform has been successfully initialized!

Actual Behavior

Terraform Init First Run

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/archive v2.0.0...
- Installed hashicorp/archive v2.0.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.20.0...
- Installed hashicorp/aws v3.20.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Terraform init Second run

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/archive v2.0.0...
- Installed hashicorp/archive v2.0.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.20.0...
- Installed hashicorp/aws v3.20.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Relevant Configuration

terraform {
  required_version = ">= 0.14"
  required_providers {
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.0.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.0.0"
    }
    template = {
      source  = "hashicorp/template"
      version = "~> 2.2.0"
    }
  }
}

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

terraform init (multiple times for the same configuration)

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

Terraform Init -get-plugins=false does not fail as expect in 0.13.x and beyond. #27092 (closed, associated with this problem, but not the same use case)

The text was updated successfully, but these errors were encountered:

adicostil · 2020-12-06T19:17:47Z

This behavior seems to be by design.

to solve the caching issue you mentioned you need to tell terraform to cache the plugins. Add this line in $HOME/.terraformrc

  plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"

then it should work as expected. Not sure why this is not the default behavior.

see https://www.terraform.io/docs/commands/cli-config.html

joncolby · 2020-12-06T20:01:35Z

@adicostil - thanks for this! This helps to "workaround" the problem seen with terraform 0.14.1

I tested this with terraform 0.13.5 and the providers download behaviour is different (it is the expected behaviour)

This is the Terraform 0.13.5 output after running terraform init a second time:


Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.20.0
- terraform.io/builtin/terraform is built in to Terraform
- Using previously-installed hashicorp/archive v2.0.0
- Using previously-installed hashicorp/local v2.0.0
- Using previously-installed hashicorp/null v3.0.0
- Using previously-installed hashicorp/external v2.0.0
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed hashicorp/kubernetes v1.13.3
- Using previously-installed terraform-providers/statuscake v1.0.0
- Using previously-installed carlpett/sops v0.5.3
- Using previously-installed hashicorp/random v3.0.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

apparentlymart · 2020-12-07T23:51:08Z

Hi all,

What's happened here is that previously Terraform was using the contents of the .terraform directory as a sort of implied dependency lock -- the presence of a provider in there caused Terraform to re-select that version unless you use -upgrade -- but now the .terraform.lock.hcl file has replaced it, so Terraform only considers the lock file when deciding what to install, and it does not consider what's already installed in .terraform because the lock file is always authoritative.

Of course, a side-effect of that new design is that the lock file can only specify a version, and can't also serve as a sort of implied "cache" of the packages. Therefore Terraform always installs what the lock file says because the lock file doesn't contain any packages.

It would be possible in principle for Terraform to treat the .terraform directory just as a cache, with .terraform.lock.hcl serving as the authority for which packages are needed. That's some new behavior that we'll need to design, so I'm going to relabel this as an enhancement just to send this into our "design a new behavior" process, rather than our "make the implementation match the intended design" (bug fixing) process.

joncolby · 2020-12-08T13:20:45Z

@apparentlymart - Thanks for the explanation. Now I understand how this behaviour relates to the new lock mechanism.

For now, I will use the plugins-cache-dir as a workaround to achieve the desired cache behaviour.

keep up the good work!

erezool · 2020-12-08T21:26:28Z

@apparentlymart - Thanks for the explanation. Now I understand how this behaviour relates to the new lock mechanism.

For now, I will use the plugins-cache-dir as a workaround to achieve the desired cache behaviour.

keep up the good work!

Only that the offered work around doesn't work too.

joncolby · 2020-12-10T20:30:53Z

@erezool - The workaround works for me. I am using a client configuration file .terraformrc containing the following. what did you try exactly?

plugin_cache_dir = "mycachedir"

erezool · 2020-12-10T21:34:26Z

@joncolby Thanks for replying. It does work now and I didn't change a thing. Pretty strange I must say

apparentlymart · 2021-10-15T22:00:18Z

Hi all!

It looks like this issue was duplicated with #27534, but we didn't notice at the time. We resolved that in v0.14.6 so that Terraform's installer will now only attempt to fetch and install a provider package if the version indicated in the lock file isn't already present. That is, in effect, what I described in my earlier comment above.

Given that, I'm going to close this. If you are still seeing redundant installation on Terraform v0.14.6 or later then please open a new issue to discuss that, since the details of how the installer works are now different than they were for this issue and so we'll need to start over with a new set of debug/trace logs, etc, in order to investigate.

Thanks!

github-actions · 2021-11-15T02:05:46Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

joncolby added bug new new issue not yet triaged labels Dec 5, 2020

joncolby changed the title ~~0.14.0 terraform init redownloads plugins every time (reopen request)~~ terraform init redownloads plugins every time (reopen request) Dec 6, 2020

joncolby changed the title ~~terraform init redownloads plugins every time (reopen request)~~ 0.14.0 terraform init redownloads plugins every time (reopen request) Dec 6, 2020

danieldreier added the v0.14 Issues (primarily bugs) reported against v0.14 releases label Dec 7, 2020

apparentlymart added cli enhancement and removed bug new new issue not yet triaged v0.14 Issues (primarily bugs) reported against v0.14 releases labels Dec 7, 2020

apparentlymart changed the title ~~0.14.0 terraform init redownloads plugins every time (reopen request)~~ Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory Dec 8, 2020

aliscott mentioned this issue Dec 16, 2020

Skip init if it has already run infracost/infracost#274

Closed

apparentlymart closed this as completed Oct 15, 2021

github-actions bot locked as resolved and limited conversation to collaborators Nov 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory #27155

Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory #27155

joncolby commented Dec 5, 2020 •

edited

Loading

adicostil commented Dec 6, 2020

joncolby commented Dec 6, 2020 •

edited

Loading

apparentlymart commented Dec 7, 2020

joncolby commented Dec 8, 2020

erezool commented Dec 8, 2020

joncolby commented Dec 10, 2020

erezool commented Dec 10, 2020

apparentlymart commented Oct 15, 2021

github-actions bot commented Nov 15, 2021

Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory #27155

Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory #27155

Comments

joncolby commented Dec 5, 2020 • edited Loading

Terraform Version

Expected Behavior

Actual Behavior

Relevant Configuration

Steps to Reproduce

References

adicostil commented Dec 6, 2020

joncolby commented Dec 6, 2020 • edited Loading

apparentlymart commented Dec 7, 2020

joncolby commented Dec 8, 2020

erezool commented Dec 8, 2020

joncolby commented Dec 10, 2020

erezool commented Dec 10, 2020

apparentlymart commented Oct 15, 2021

github-actions bot commented Nov 15, 2021

joncolby commented Dec 5, 2020 •

edited

Loading

joncolby commented Dec 6, 2020 •

edited

Loading