Cannot use terraform import with module that has dynamic provider configuration

[jszwedko]

Terraform Version

Terraform v0.9.1

Affected Resource(s)

N/A

Terraform Configuration Files

# ./main.tf
module "module_us_west_1" {
  source = "./module"
  region = "us-west-1"
}

# ./module/main.tf
variable "region" {
  description = "AWS region for provider"
}

provider "aws" {
  region = "${var.region}"
}

resource "aws_cloudwatch_log_group" "rds_os" {
  name = "RDSOSMetrics"
  retention_in_days = 30
}

Debug Output

https://gist.github.com/ff54870fee49636209ecfaa5de272175

Panic Output

N/A

Expected Behavior

Resource was imported

Actual Behavior

Error importing: 1 error(s) occurred:

* module.module_us_west_1.provider.aws: 1:3: unknown variable accessed: var.region in:

${var.region}

Steps to Reproduce

1. terraform import module.module_us_west_1.aws_cloudwatch_log_group.rds_os "arn:aws:logs:us-west-1:FILTERED:log-group:RDSOSMetrics:*"

Important Factoids

N/A

References

N/A

[Flygsand]

Same issue here on v0.9.2. Seems related to #7774 (closed).

[iancward]

I'm seeing the same issue with the datadog provider, so this isn't just AWS.

[haidangwa]

I've found that adding a provider alias resolves this issue. Something static works, like alias "myregion".

[iancward]

It looks like you can sort of get around this by aliasing the provider.

provider "aws" {
  alias = "${var.region}"
  region = "${var.region}"
}

resource "aws_cloudwatch_log_group" "rds_os" {
  provider = "aws.${var.region}"
  name = "RDSOSMetrics"
  retention_in_days = 30
}

This may also be worth a read: #1819
As is this: #3285

[Chili-Man]

I'm having the same issue as well with Terraform 0.9.4

[bflad]

Seeing this behavior with Terraform 0.9.6 and a similar use case of a module with a variable in the provider:

# module/variables.tf
variable "aws_profile" {
  description = "AWS profile for the provider"
  type        = "string"
}

variable "aws_region" {
  description = "AWS region for the provider"
  type        = "string"
}

# module/provider.tf
provider "aws" {
  profile = "${var.aws_profile}"
  region  = "${var.aws_region}"
}

* module.EXAMPLE.provider.aws: 1:3: unknown variable accessed: var.aws_profile in:

${var.aws_profile}

[christianbradley]

This behavior still exists in v0.10.3. Any update on this?

[apparentlymart]

Hi all! Sorry for this limitation and for the long silence here.

Unfortunately the import system struggles a bit with more complex situations like this because it builds a different sort of graph to deal with the import case.

From the example config and output given here, it looks like the interpolation context isn't being constructed correctly when in import mode, and so the variables from the parent module aren't coming through correctly. This is definitely a bug, but is likely fiddly to fix. 😖

The fact that aliasing the provider makes this work is interesting. In that situation, is there an unaliased provider "aws" block (with a literal region) in the parent/root module that could be being used instead? That's the best explanation I can come up with for why that didn't produce an error that the region needs to be set.

[christianbradley]

@apparentlymart - sounds fiddly indeed. Is there anyone assigned to this bug who can get fiddlin?

[apparentlymart]

There isn't anyone available to look at this right now, but want to get to it eventually. Any additional information we can gather in the mean time (including the question at the end of my previous comment) could help explain the issue here and thus make it easier to plan about.

To be honest, the current import functionality is primarily focused on simple cases where people are getting started and importing for the first time, so its design struggles with more complex scenarios. We will probably end up having to rethink it in a broader sense before too long in order to make it more usable.

Our usual expectation is that import is something people use only for a short period when they are getting started with Terraform, but from the comments here I get the sense that some or all of you are using it in a more ongoing/routine manner. Is that right? If so, it'd be great to hear what you're using it for since that will help inform future changes to make it more generally-usable.

[christianbradley]

@apparentlymart - I can't answer your question personally as the "workaround" didn't work for me...

I won't say I'm a terraform expert just yet - I've been using "import" to migrate legacy deploys of an app we develop into terraform control so that future updates can be automated with terraform.

Would you suggest a different/faster method for pulling existing resources into state so that terraform does not try to destroy/recreate them?

[apparentlymart]

I've not personally run into this since I've only used import in pretty simple cases, but when faced with this problem I would probably try to work around it like this:

• Import the resource into the root module, using a provider that has the intended region statically configured.
• Use terraform state mv to move the state for the imported resource over into the target module.
• Move the configuration block into the target module too.
• Run terraform plan to make sure things have settled.

I'm not under any illusion that the above is a great workaround, but I think it would work and what I'd try when faced with is problem right now.

If the root module's config is explicitly using a different region, a variant of the above would be to temporarily create a child module that exists only to receive imports, with a static provider config, and import into that before moving into the final location.

To be completely honest, I'm not sure why the workaround of temporarily adding an alias works, so I would not have thought to try that. If it is working, I expect it's due to some edge case and thus not something I'd expect to work reliably.

[grahamlyons]

@iancward's alias workaround worked for me.

[philax]

the alias workaround is not working for me

[vascop]

We're using dynamic configuration of modules to be able to store our secrets (postgresql password, datadog API keys...) in Vault and pull them to be used by terraform when importing, planning and applying. Hinting that we should just give everyone passwords for everything and risk them be committed to source control doesn't look like sound advice.

Furthermore, this happens any time we're importing anything, even from another provider. So if I want to start using Terraform to provision a new provider, I have to go to my other providers and replace all the dynamic references to static ones again.

[wraithm]

I did exactly what @apparentlymart suggested here. What happened is that the terraform plan completely plowed over my carefully constructed terraform state. I think that it's actually only terraform refresh that's causing the state to be changed.

In the root module, I have an aws provider that is unaliased. It seems that in my modules where I use different providers, associated with different aws accounts, the resources in there got replaced with resources associated with that was provider that is in the root module.

EDIT: It was only terraform refresh that destroyed state. terraform plan works perfectly fine. Aliasing the unaliased provider after doing all of the imports and moves makes terraform refresh not destroy state.

[daveadams]

I have a slightly different concern about this problem, which has gotten worse with 0.10 changes to how imports work. In order for the workaround of importing to the root module, then doing terraform state mv, you also need to create temporary resource stubs in the code. This has practical implications that make the import process truly burdensome.

My understanding is that the requirement to stub out the imported resources was imposed to prevent people from then immediately accidentally destroying those imported resources by running terraform apply. But with the new defaults that default to requiring interactive approval before application, this is surely a much smaller concern.

As far as the provider config, I'm also willing to specify all of that on the command line. So it'd be great if we had the option to ignore the resource and provider config in the tf code, and just specify the provider args on the command line, and give a path we wish to import to.

This is becoming a bigger problem for us as we try to get more people in our organization to use Terraform. The lack of ability to import into any but the most explicitly specified code is a real block to adoption.

[sebglon]

this works good for me on v0.10.7 with Google provider.

[christianbradley]

This is really becoming a major issue for adopting terraform for our current app. It is composed of multiple modules (vpc, dns, auth, api, cms, portal, etc) that have dependencies upon one another. This is very easily definable using a root module that passes output vars of one module to the input vars of another, but then we lose the ability to import existing infrastructure (which is a common scenario for some of our legacy apps). The alternative is to write and maintain a ton of scripts that do the same thing while keeping each module independent.

The response from the terraform dev team isn't giving us a lot of faith in the platform either, it's been 10 months since this issue was opened and we've seen very little actual focus on it. We're currently investigating alternatives to terraform as a result.

I'd love to keep using terraform, but as the weeks go by I'm seeing many such pain points - and it's getting harder to convince the rest of the team to keep moving forward.

@apparentlymart - it's been 3 months since we've had any update from the team on this issue, other than saying "it's fiddly to fix". Some bugs are fiddly, I get it. But these are two of the features that drew us to terraform...

1. Modular definitions
2. The ability to import existing infrastructure into TF management.

Let's get this fixed, or honestly, we're moving on.

[leszekeljasz]

Suggested solution with aliases does not solve the problem for me. It fixes one, and adds another - terraform keeps asking for provider.aws.region each time I run import / plan / apply / destroy.

Terraform version:

$ terraform version
Terraform v0.11.2
+ provider.aws v1.7.1

Ex. config:

$ tree
.
├── main.tf
└── modules
    └── environment
        └── env.tf

main.tf

$ cat main.tf
module "uswest" {
    source  = "modules/environment"
    sg_name = "test_sg_uswest"
    region  = "us-west-2"  # Oregon
}

module "useast" {
    source  = "modules/environment"
    sg_name = "test_sg_useast"
    region  = "us-east-1"  # N. Virginia
}

modules/environment/env.tf

$ cat modules/environment/env.tf
variable "region" {}
variable "sg_name" {}

provider "aws" {
    alias   = "${var.region}"
    region  = "${var.region}"
}

resource "aws_security_group" "sg" {
    name = "${var.sg_name}"

terraform plan

$ terraform plan
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Default: us-east-1
  Enter a value:

[philax]

Is there any sort of ETA on this at all?

[meowmeowhappykitten]

My hacky workaround for this (that probably won't work in more complex situations) is just to fill in ${var.region} 's in my module with the actual string value of the region while I'm importing a resource from. After the import I put the ${var.region} reference back in and I can verify via terraform plan that my resource has successfully been imported.

This works for me because my main.tf files only refer to one region. If you had a main.tf with the same module creating resources in multiple regions, this work around probably wouldn't work for you.

[msmagoo87]

Encountered in v0.12.23. Please fix!

[renatoargh]

Encountered in v0.12.24

[allymat]

Seems like this is still a big issue for lots of people - come on GitHub team, get this prioritized!

[kzap]

3 years and counting.... note to self if you find this issue again in 1 year don't worry its not your fault, add +1 to yearly counter

[llamahunter]

So, https://www.terraform.io/docs/commands/import.html#provider-configuration says that provider configuration during import can depend on variables (but not data sources). yet this does not work if the provider is in a module.

[abrahamoshel]

So, https://www.terraform.io/docs/commands/import.html#provider-configuration says that provider configuration during import can depend on variables (but not data sources). yet this does not work if the provider is in a module.

Just ran into the same problem. I am using a module that has out puts. It runs the plan and the apply perfectly but it does not allow me to import or rm state.

[KursLabIgor]

this topic and info helped me solve import issue
https://medium.com/@lhk.igor/this-is-how-i-solved-problem-of-importing-resources-in-terraform-with-dynamic-provider-a2f255a9a303

[mhenniges]

I faced this issue as well, and after reading this thread and all the workarounds, and decided to first just try commenting basically all of my other terraform out before importing. The import worked perfectly, and after uncommenting everything again, was back to a clean plan.

[spikewang]

Is there a way to manually create the state for the resource if it's not even related to providers?

I have the provider dynamically set in the module as well, but trying to import an existing kubernetes namespace... it's still failed on aws provider... which doesn't make any senses, since it's not even related...

[archoversight]

This is an issue even when it is not the provider that is dynamically built, but instead if you have a module that contains dynamic variables that is generated in an upstream module and passed through as a variable to a downstream module it fails to import because the downstream never receives the upstream data correctly.

This makes it very difficult to import the resource.

Similar to this comment: #13018 (comment)

[brikis98]

We ran into this issue as well. Any time we had a module with a provider block nested in it, and that provider block used any dynamic data (e.g., set region to a variable that was passed in), import would no longer work.

I just updated Terragrunt with a new aws-provider-patch command, which is an experimental, temporary workaround for this issue. You can run a command like this:

terragrunt aws-provider-patch --terragrunt-override-attr region=eu-west-1

And Terragrunt will:

1. Run terraform init to download the code for all your modules into .terraform/modules.
2. Scan all the Terraform code in .terraform/modules, find AWS provider blocks, and hard-code the region param
   to eu-west-1 for each one. You can of course set other params to override using the --terragrunt-override-attr option.

Once you do this, you'll hopefully be able to run import on that module. After that, you can delete the modified .terraform/modules and go back to normal.

Note that you should be able to use this even if you're not a Terragrunt user. Just add an empty terragrunt.hcl to any Terraform module you're using (e.g., run touch terragrunt.hcl), and run the same terragrunt aws-provider-patch command as above.

This is obviously an ugly, hacky solution, but as this bug has been open for ~3 years now, I figured an ugly solution is better than none. The fix is available in Terragrunt v0.23.40 and above. Give it a try and let me know if it works! PRs to improve it further are also welcome.

[JohannesAtGit]

@brikis98
This terragrunt aws-provider-patch worked for us fine. Thank you for this, but we have a similar problem in Azure!
Is there any plan to fix this in Terraform itself?

[brikis98]

@JohannesAtGit We aren't currently focused on Azure, so it's hard to carve out time for that right now, but if someone wants to make aws-provider-patch more generic (e.g., provider-patch), we'd very much welcome PRs!

[wagnst]

Any update on this? Still facing in 0.14

[jbardin]

Hello All!

Sorry about the delay, but I have confirmed that the original issue as presented here now works in the latest release.

Since the original examples here are strictly concerned with using a provider which obtains its configuration via module variables, and those cases presented are now working in the current development release, I'm going to close this out. A lot has changed in Terraform over the course of this thread, and it's becoming difficult to follow which parts here could be relevant to the current version of Terraform.

Note that now we no longer recommend provider configurations within modules, and have better methods for explicitly passing provider configuration into modules. This form of provider configuration will also work with import when adapted to the configurations presented here.

As stated in the docs, it's still not possible to use a data source which is not already present in the state for provider configuration during import. This is a known shortcoming, and I'm going to open a new issue specifically for this concern.

If there are other examples where providers are not configured properly during import, please test with the latest development release and file a new issue, or ask in the community forum where there are more people ready to help.

Thanks!

[llamahunter]

Still not working with terraform 0.14.7

% terraform import kubernetes_config_map.qea1 kafka-copier-core/qea1-backup-topics
Acquiring state lock. This may take a few moments...

Error: Invalid provider configuration

  on /Users/rdlee/Documents/tivo-research/git/inception-k8s-deployment-dev/backup/tek2/main.tf line 58:
  58: provider "kubernetes" {

The configuration for provider["registry.terraform.io/hashicorp/kubernetes"]
depends on values that cannot be determined until apply.


Error: Invalid provider configuration

  on /Users/rdlee/Documents/tivo-research/git/inception-k8s-deployment-dev/backup/tek2/main.tf line 64:
  64: provider "k8sraw" {

The configuration for provider["terraform.tivo.com/nabancard/k8sraw"] depends
on values that cannot be determined until apply.

Releasing state lock. This may take a few moments...

These providers are at the root level, not in a module.

[llamahunter]

see #27934 for an update on how to fix this.