Use provider plugins directly from plugin dir / allow junctions in .terraform

[c33s]

in past terraform versions (0.11.?) terraform downloaded the plugins into the working directory but if they were manually moved to Roaming/terraform.d/plugins a terraform init or terraform plan ran successfull by using the "globally installed plugins.

with terraform 0.13.2 (or maybe even 0.12.x) this possibility is gone, the plugins are always copied to .terraform/plugins in the working directory.

according the issue #16262 this behavior is intended.

The behavior when caching is enabled is that Terraform should download the plugins into the cache directory and then create a link (hardlink if possible, symlink otherwise)

@apparentlymart #16262 (comment)

feature request: please bring back the old behavior where having the plugin in the Roaming/terraform.d/plugins directory is directly used.

the reason i also raised this as bug and security issue is that we use very strict firewall rules. so each plugin has it's own firewall rules. the firewall is best handled if the plugins doesn't change their name (terraform-provider-external.exe no version string attached) in this case the firewall would only ask if it was ok that the hash changed for the file and automatically applies the old rules for the new file. of course this is not possible with terraform as it has to keep different versions in parallel.
it's ok to manage the rules for each plugin in each version once but now as the plugins are copied to a new location they are "new" files for the firewall which need their own rules. so now it's one rule for each plugin version for each project which is really a lot of overhead.

our workaround in such cases is to use a junction on windows (kind of symlinks which does not require the special admin privilege) but symlinking (junction) is also prevented by terraform:

Plugin reinitialization required. Please run "terraform init".

why? 😢

so the only thing is left to lower the config overhead is that we would have to disable the firewall for the whole outgoing traffic. this is not an option.

can you please bring back the old behavior or at least allow us to use directory junctions? hardlinks are not an option as they don't work accross different partitions and symlinks are also quite unhandy as they require a special permission to be created.

[jbardin]

Hi @c33s,

I understand the need to have a configurable location for provider storage, but I'm not sure what you're missing from the current implementations. I assume the filesystem_mirror is what you're looking for (which has some default locations), and there are other options under the Provider Installation section of the configuration docs if that does not suffice.

Is there a problem using the filesystem_mirror or plugin_cache_dir with a filesystem junction?

[c33s]

@jbardin i hope this time my explanation of the problem is better:

lets assume we have a project with the following code in the directory C:\example-project

main.tf:

terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
      version = "1.20.1"
    }
  }
}

provider "hcloud" {
  token = "${var.hcloud_token}"
  version = "1.20.1"
}

and the following code in C:\Users\username\AppData\Roaming\terraform.rc

plugin_cache_dir   = "$APPDATA/terraform.d/plugin-cache"

in terraform 0.11 and 0.12 running terraform init will create

• C:\Users\username\AppData\Roaming\terraform.d\plugin-cache\windows_amd64\terraform-provider-hcloud_v1.20.1.exe
• C:\example-project\.terraform\plugins\windows_amd64\terraform-provider-hcloud_v1.20.1.exe

if i move terraform-provider-hcloud_v1.20.1.exe from the project dir .terraform\plugins\windows_amd64 to
C:\Users\username\AppData\Roaming\terraform.d\plugins and keep the lock file in place, the next terraform init will do
nothing (nothing will be downloaded or copied) and on calling terraform plan the shared plugin from the AppData dir
is used (works for all projects).

with terraform 0.13 the behavior has changed. running terraform init will create

• C:\Users\username\AppData\Roaming\terraform.d\plugin-cache\registry.terraform.io\hetznercloud\hcloud\1.20.1\windows_amd64\terraform-provider-hcloud_v1.20.1.exe
• C:\example-project\.terraform\plugins\registry.terraform.io\hetznercloud\hcloud\1.20.1\windows_amd64\terraform-provider-hcloud_v1.20.1.exe

if i again move the content from C:\example-project\.terraform\plugins to C:\Users\username\AppData\Roaming\terraform.d\plugins
the next terraform init will recreate the plugin dir in the project directory. of course it will not download it again
but i cannot prevent terraform from recreating the exe file in the project directory. which of course leads to a, for the
firewall, new file as it is on a new location and not called from a known location (the C:\example-project\.terraform\plugins dir).

as far as i understand filesystem_mirror, it won't help as it's implicit configured already.

The implied configuration includes a selection of filesystem_mirror methods and then the direct method.

for my understandings this means if the plugin is not cached already do a "direct" download and the copy it to the project
directory or if already cached do a filesystem_mirror which again means copy it to the project directory (haven't tried
to explicit configure the filesystem_mirror but i would like to use the implicit config.

also the name mirror says "i copy/mirror the plugins to your project directory" and not "i call the plugins direct from
the C:\Users\username\AppData\Roaming\terraform.d\plugins or the plugin-cache directory" which would be the behavior we
required and had in the previous terraform versions.

the docs also state that the working directory (project dir) is self-contained, which is exactly what we don't want,
we want the "shared plugin" behavior.

By default, terraform init downloads plugins into a subdirectory of the working directory so that each working directory is self-contained.

as described above we tried to manually link the shared directory to the project diretory

C:\example-project\.terraform\plugins\registry.terraform.io
->
C:\Users\username\AppData\Roaming\terraform.d\plugins\registry.terraform.io

but this leads to

Error: Failed to install provider from shared cache

Error while importing hetznercloud/hcloud v1.20.1 from the shared cache
directory: after linking registry.terraform.io/hetznercloud/hcloud from
provider cache at C:\Users\username\AppData\Roaming/terraform.d/plugin-cache
it is still not detected in the target directory; this is a bug in Terraform.

but we don't want to install "from" shared plugin dir we would like to "use" them from the shared plugin dir.

we tried to link at multiple depths:

C:\Users\username\AppData\Roaming\terraform.d\plugins\registry.terraform.io\hetznercloud\hcloud\1.20.1\windows_amd64\

• plugins
• registry.terraform.io
• hetznercloud\hcloud
• 1.20.1
• windows_amd64

every depth doesn't work only windows_amd64 let terraform init pass.

what we would need would be another option like shared, where the plugins are used and not copied from the shared plugin
directory.

i hope my expanation was clearer this time.

[jbardin]

Thanks for the clarification @c33s, I see what you are trying to do now.

It is my understanding that we cannot use junctions directly in terraform due to the default behaviors of windows resulting in the accidental deleting of the binaries along with temporary directories. Terraform will attempt to make a symlink of the provider, and when that's not possible it will copy the provider into the working location.

There are no current plans to change this behavior, and the recommended method to prevent the copy would be to provide the user with the correct permissions to create a symlink. It's not entirely clear why manually linking the directories did not work however, and that could be looked into.

[c33s]

@jbardin directory junctions are quite safe to use, if you delete a junction the original directory is not touched (keywords are junction and directory here). real symlinks need the akward permission set by the admin or needs you to work as admin (both are security issues (something only needs a special explicit allowed permission if you can do something bad with it. you need no extra permission for a directory junction)).

i don't know the behavior in "go" but in "php" you have to use rmdir to delete a junction and not unlink. yes windows is different but the brute force copy method is a real security problem on hardened systems so i hope you rethink your position of not changing the current behavior.

also why the copy at all it the files are already in the shared dir? this is also a space problem. if you have a very fast ssd, it's often quite expensive and therefore quite small, so it's also a space problem if everything is duplicated instead of using it from a single location.

is it such a huge loss if you delete the plugin binaries by mistake (should it ever happen)? as they can easily be recovered from the registry. it's like deleting the vendor dir in php or the node_modules dir in javascript.

not changing the behavior to the old one means a real security impact as

• each plugin has to be allowed (setup firewall rules) manually for each project in each version.
• the firewall has to be disabled
• an extra permission for symlinking has to be granted

from my point of view, only because nobody (not many people) care about security, this should still be handled as security issue, even if nobody cares and simply work as admin, disable a firewall or grant higher permissions without even knowing the sideeffects.

you as tool provider for operations which handle many many server should care about security and allow hardend systems. shouldn't it be your responsibiility to provide such a security allowing tool even if most of the people don't use the security feature?

[Phydeauxman]

I too am trying to use the filesystem_mirror feature in .terraformrc on a Ubuntu system and it keeps creating hard links in the project folder instead of symlinks. How do I get it to create symlinks to reduce the size of our repo?

[pkyeck]

using export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache" on macOS works like a charme. Downloads plugins once and symlinks them into the project-folder