Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform State Snapshot Locked to 0.12.13 After Apply #23290

Open
enummela opened this issue Nov 5, 2019 · 57 comments
Open

Terraform State Snapshot Locked to 0.12.13 After Apply #23290

enummela opened this issue Nov 5, 2019 · 57 comments

Comments

@enummela
Copy link

enummela commented Nov 5, 2019

I upgraded my terraform CLI from 0.12.9 to 0.12.13 while attempting to fix a problem unrelated to this bug report. In the process of dubugging this problem, I ran terraform apply unsuspecting that it would seemingly (for all intents and purposes) break my state in 0.12.9 forever.

It turns out my problem was not solved by upgrading my CLI to 0.12.13 so I downgraded it back to 0.12.9. However, now in my Terraform project and all projects that pull in its state remotely I am experiencing this error on plan and apply.

Error: state snapshot was created by Terraform v0.12.13, which is newer than current v0.12.9; upgrade to Terraform v0.12.13 or greater to work with this state

Terraform Configuration Files

...

terraform {
  backend "s3" {
    bucket         = "my-bucket"
    region         = "us-west-2"
    key            = "eks-terraform/state"
    dynamodb_table = "my-dynamodb-table"
    encrypt        = true
  }
}

...

Expected Behavior

After upgrading to 0.12.13, on running terraform apply a prominent message is displayed warning me that continuing will upgrade my state and that it will be unusable in older versions of Terraform. Ideally, I would also have to provide some kind of flag or argument to bypass this warning so that I don't accidentally lock my state into this newer release.

Alternatively, some kind of mechanism will exist allowing me to revert my state snapshot to a format that's compatible with the older release (0.12.9).

Actual Behavior

My Terraform project is now dependent on 0.12.13 so now all of my team members and I have to upgrade our CLIs and upgrade the CLI in our CI pipeline Docker image.

@enummela enummela changed the title Cannot Back Out of Terraform Upgrade After Apply Terraform State Snapshot Locked to 0.12.13 After Apply Nov 5, 2019
@hposca
Copy link

hposca commented Nov 9, 2019

I'm also getting a Error: state snapshot was created by Terraform v0.12.13, which is newer than current v0.12.8; upgrade to Terraform v0.12.13 or greater to work with this state message.

And my scenario is even stranger... My terraform executable and state file are still on version 0.12.8:

$ terraform --version
Terraform v0.12.8

$ terraform state pull | grep terraform_version
  "terraform_version": "0.12.8",

It also uses S3 as backend, and I believe that nobody in the company did a terraform apply (0.12.13) as terraform_version is still 0.12.8 on the state file.

We are using 0.12.13 on other completely different resources, with their respective state files. (I don't see how, but) Is there a chance of "cross-contamination" (via the DynamoDB table or something else)?

P.S.: We are using tfenv to run different versions of terraform depending on what we are building.

@dogzzdogzz
Copy link

same issue here, we have teams in terraform collaboration, i don't want to be forced to upgrade my terraform version frequently just because someone use a newer version but actually there is no breaking change in tfstate between two versions.

@hposca
Copy link

hposca commented Dec 18, 2019

This happened again today.

Error: state snapshot was created by Terraform v0.12.18, which is newer than current v0.12.17; upgrade to Terraform v0.12.18 or greater to work with this state

$ terraform --version
Terraform v0.12.17

$ terraform state pull | grep terraform_version
  "terraform_version": "0.12.17",

@hoangnguyenba
Copy link

This happens again today :(
Error refreshing state: state snapshot was created by Terraform v0.12.18, which is newer than current v0.12.16; upgrade to Terraform v0.12.18 or greater to work with this state

`terraform --version
2019/12/31 11:58:12 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.

2019/12/31 11:58:12 Loading CLI configuration from /Users/heakthone3/.terraformrc
Terraform v0.12.18

  • provider.acme v1.5.0
  • provider.alicloud v1.67.0
  • provider.digitalocean v1.12.0
  • provider.helm v0.10.4
  • provider.kubernetes v1.10.0
  • provider.tls v2.1.1`

@hoangnguyenba
Copy link

In my case, I need to go remote state and update the version to 0.12.18

@voldemortensen
Copy link

This happened to me today. I imported a resource from Kubernetes, which updated the state. State is kept in Terraform Cloud. terraform plan won't complete because of this error. It seems that Terraform Cloud hasn't updated the system that runs the plans.

@voldemortensen
Copy link

In case anyone else needs it, I stumbled upon this in Terraform Cloud in Settings > General.

The version of Terraform to use for this workspace. Upon creating this workspace, the latest version was selected and will be used until it is changed manually. It will not upgrade automatically.

Changing this resolved my issue. Very inconvenient, but that's the solution.

@ns0092
Copy link

ns0092 commented Feb 3, 2020

I had this same issue.
If you see this error, its not just the version of the terraform state that you are accessing/running plan against. Terraform cross-references a lot of terraform states internally. So just go inside the remote state bucket and try to find that one specific remote state with different tf version.

In my case, I was running a tf plan for resource 'X' that clearly had the remote state version 0.12.8. But resource 'X' was global and it was referencing another totally different resource Y's state with 0.12.14 which is why, I was getting the error. Just open the bad remote state in a text editor and change the "terraform_version" value to the default version that you want. This will fix it 👍

@msotoperdomo
Copy link

Same issue here. Getting a plan error for when the remote state version is greater than the state that uses it.

error loading the remote state: state snapshot was created by Terraform
v0.12.13, which is newer than current v0.12.12; upgrade to Terraform v0.12.13
or greater to work with this state

I use the same remote state in a terraform version 0.11.14 and this validation is not present. Plan works fine.

@mike-dazn
Copy link

Still happening on 0.12.23/0.12.24

Error: state snapshot was created by Terraform v0.12.24, which is newer than current v0.12.23; upgrade to Terraform v0.12.24 or greater to work with this state

With this issue, it makes remote state an absolute nightmare to manage.

@garrypas
Copy link

garrypas commented May 7, 2020

This bug needs to be fixed, causes huge amounts of hassle. Upgrading isn't always an option, and trying to aggressively force people down the upgrade path is wrong. People use the wrong version and the state is basically screwed, I've never seen an effective solution to this other than spending days digging into buckets to remove residual crap that terraform refuses to destroy by hand and start afresh.

@baumstern
Copy link

Still happening on 0.12.21/0.12.16

Error refreshing state: state snapshot was created by Terraform v0.12.21, which is newer than current v0.12.16; upgrade to Terraform v0.12.21 or greater to work with this state

@etwillbefine
Copy link

I have the same issue with Terraform 0.12.24. Error: state snapshot was created by Terraform v0.12.24, which is newer than current v0.12.19.

This happens to me once an autoplan was triggered and failed and I run atlantis plan again. The default tf version and project config are both set to 0.12.24 but the binary installed on the docker image used version 0.12.19. So somehow running atlantis plan after a plan was already created causes the "local binary" (shipped with the docker image) to be used.

@Jangmin-Lee
Copy link

I have the same issue with 0.12.20 -> 0.12.24
Error refreshing state: state snapshot was created by Terraform v0.12.24, which is newer than current v0.12.20; upgrade to Terraform v0.12.24 or greater to work with this state

@alencarlucas
Copy link

It still occurring with 0.12.24 - 0.12.25
Is there anyone looking at this? I think that kind of breaking changes should be avoided in minor versions releases.

@jltxwesley
Copy link

In case anyone else needs it, I stumbled upon this in Terraform Cloud in Settings > General.

The version of Terraform to use for this workspace. Upon creating this workspace, the latest version was selected and will be used until it is changed manually. It will not upgrade automatically.

Changing this resolved my issue. Very inconvenient, but that's the solution.

Nice, this works perfectly.

@mikesigs
Copy link

mikesigs commented Jun 2, 2020

I'm just here to throw my voice into this apparent echo chamber of an issue. This is super annoying! Can this please get fixed somehow?

@ns0092
Copy link

ns0092 commented Jun 2, 2020

This happens when more than 1 person is running tf apply and their terraform binary version is not the same. You may have terraform 0.12.8 while the other guy may have 0.12.20. In that case, once that person runs tf apply, the version on terraform remote state will be set as 0.12.20 and you will see the above error saying your version doesn't match the one existing on remote state.

To fix this, you can open the terraform remote state (its basically a json) and update the value of terraform version to a specific version you want. Then make everyone on your team use the same version terraform binary using the link: https://releases.hashicorp.com/terraform/

To avoid this, you can also pin the version on your tf code as

terraform {
   required version = "=0.12.20"
}

@dkirrane
Copy link

dkirrane commented Jun 11, 2020

Any workaround for this on Terraform Cloud?
In my Terraform Cloud workspace I have Settings > General > Terraform Version > 0.12.24

My Plan uses:

terraform {
  required_version = "= 0.12.24"
}

My TFC workspace State tab has no state: This workspace currently has no states saved

When I queue the plan I hit this:
Error: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state

The issue occurred because the initial workspace run that used v0.12.26. I swicthed to 0.12.24 and this started happening

@JeremyCraigMartinez
Copy link

@dkirrane I had this exact version discrepancy. I think since the versions are so close, they are probably on the same version of tfstate json. For this reason, I pulled the tfstate from S3 (storing state remotely) and simply changed the "terraform_version": "0.12.26", back to "terraform_version": "0.12.24", (second line down in the json), then reuploaded it to the same S3 key.

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 10,
  ...

I think this is (very) ill-advised, but it seems to have done the trick for me 🤷‍♂️ . It's also reassuring that I was making this change in a non-production environment.

@rozcietrzewiacz
Copy link

Dear Hashicorp,

Why is this tagged as enhancement instead of bug? This is clearly a significant design flaw that is breaking workflow and adding a lot of unnecessary work for people.

If the rationale behind that "feature" is to prevent backwards-incompatible changes breaking the use of an older (patch!) version, then I'm sure you can do better than this.

@mike-dazn
Copy link

delete your .tfstate in your bucket,
and try again to terraform init

That is really poor advice and is a fundamental misunderstanding of what remote .tfstate is for. Anyone who does this risks seriously damaging their infrastructure setup.

@abhisheksr01
Copy link

The behaviour is still the same with latest version :( :(

Error refreshing state: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.28; upgrade to Terraform v0.12.29 or greater to work with this state

@Ayxan1
Copy link

Ayxan1 commented Jul 28, 2020

The same problem...

Error refreshing state: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.28; upgrade to Terraform v0.12.29 or greater to work with this state

@zack-klein
Copy link

zack-klein commented Aug 1, 2020

Same problem - I was able to fix this by going into the remote .tfstate and manually changing the terraform_version, but this feels super hacky. A cleaner solution to this would be fantastic!

At first glance, a simple solution to this could be to expose a command like terraform state set-version to the CLI that just updates the .tfstate file for the user? It's not ideal but I think it would make resolving this issue once it exists much safer.

@chuck-hilyard
Copy link

Running into the same issue as others.

@estecker
Copy link

I had a similar issue when I accidentally pushed a 0.12 state with Terraform 0.13 in Terraform Cloud. That automatically set the version in the state to 0.13. I could not push an old state with 0.12 because it was older than 0.13. I was able to fix that problem via the method described here: https://support.hashicorp.com/hc/en-us/articles/360041299873

@abhineetraj
Copy link

Still having the same problem between sub-versions 0.12.28 and 0.12.26. This needs to be handled gracefully.

Error: state snapshot was created by Terraform v0.12.28, which is newer than current v0.12.26; upgrade to Terraform v0.12.28 or greater to work with this state

@dogzzdogzz
Copy link

dogzzdogzz commented Oct 19, 2020

Having same problem when trying 0.13, we have several individual environments (tfstate) and we use data.terraform_remote_state to import some data from another environment. But It's difficult for us to done the 0.13 upgrade for all environments at the same time.

@nikhilo
Copy link

nikhilo commented Dec 4, 2020

In a pretty 💩 situation right now. I created a new workspace and it automatically chose 0.14.0 on the very first plan. All the resourced showed this error.

Error: Unsupported Terraform Core version

  on .terraform/modules/rds_cluster_aurora_postgres.dns_master.this/versions.tf line 2, in terraform:
   2:   required_version = ">= 0.12.0, < 0.14.0"

Module module.rds_cluster_aurora_postgres.module.dns_master.module.this (from
git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.19.2)
does not support Terraform version 0.14.0. To proceed, either choose another
supported Terraform version or update this version constraint. Version
constraints are normally set for good reason, so updating the constraint may
lead to other errors or unexpected behavior.

Note that this was from the very first plan.
So, I tried to downgrade the version to 0.13.5
And it complains that,

Error: state snapshot was created by Terraform v0.14.0, which is newer than current v0.13.5; upgrade to Terraform v0.14.0 or greater to work with this state

There is no state file for my workspace, so I don't even know what to revert at this point.

Weirdest part is, I even deleted the workspace and recreated it. Set the version to 0.13.5 before running the first plan. And it still complains about the state snapshot was created by Terraform v0.14.0, which is newer than current v0.13.5

@Sytten
Copy link

Sytten commented Dec 5, 2020

Yeah some weird errors here too with GCP provider and 0.14.0 hashicorp/terraform-provider-google#7945, but I cant downgrade...

@nikhilo
Copy link

nikhilo commented Dec 7, 2020

To clarify the issue I mentioned earlier, the workspace I was creating refers to outputs from another workspace which was mistakenly created with 0.14.0. So that's why even when I recreated my workspace with 0.13.5, it was showing the state snapshot was created by Terraform v0.14.0 error. On recreating the other workspace with 0.13.5, the intended workspace is now working ok with 0.13.5.

@anagarjunr
Copy link

this issue still exists and is very annoying

using

  • terraform version 0.14.0
  • remote backend -> terraform cloud (using local exec mode instead of remote)

getting this error : Failed to load state: Terraform 0.14.0 does not support state version 4, please update.

while the same version 4 works for other resources in another tf-workspaces

@acidprime
Copy link

acidprime commented Jan 6, 2021

If it helps anyone else, I found an interesting corner case thats confusing due to the message displayed. I was in the process of upgrading some dependant terraform configurations (lambdas) and they use remote_state. I notice that I was getting this message , which I thought was part of the upgrade but it was in fact a chicken before the egg issue with remote_state because my current terraform version was trying to read remote state from an already upgraded version, it caused this error message , though to me it was not very clear thats where this was coming from

@giom-l
Copy link

giom-l commented Jan 12, 2021

Hit this issue today also
Upgraded from 0.13.4 to 0.14.4.
One of my stack showed lots of drifts, so I wanted to rollback to 0.13.4, but also getting
Error: state snapshot was created by Terraform v0.14.4, which is newer than current v0.13.4; upgrade to Terraform v0.14.4 or greater to work with this state

However, the state version is 0.13.4, not updated since last december (s3 backend)
I removed every cache files that could exist locally, I still don't understand where this state snapshot come from....

ATM, I have to leave it as is or accept to recreate this part of my infrastructure (as there is lots of recreate needed...)

If anyone has an idea where this snapshot version could come from or where is it located so I could change it manually, I'm very interested.

@commarla
Copy link
Contributor

@giom-l maybe (as @acidprime suggest) you have a data source terraform_remote_state already upgraded to TF 0.14.4. Terraform displays this message while trying to read a remote state within a data source (in addition to his own remote state)

@mmell
Copy link
Contributor

mmell commented Jan 12, 2021

I hit this issue too.

One of the data.terraform_remote_state modules was on 0.14.2 but the root module was on 0.12.29.

I was able to remove the reference to the data.terraform_remote_state and carry on.

@bitnahian
Copy link

Same issues here. We use a remote GCS-backend for our state. Currently facing this issue:

Failed to load state: state snapshot was created by Terraform v0.14.3, which is newer than current v0.13.5; upgrade to Terraform v0.14.3 or greater to work with this state

@scarolan
Copy link
Contributor

Please fix this way-to-easy-to-trigger footgun.

@mt-inside
Copy link

mt-inside commented Jan 30, 2021

I have a similar issue, looking to help people who hit it.
When running terraform taint I get
Failed to load state: Remote workspace Terraform version "0.12.25" does not match local Terraform version "0.14.4"

I'm using local execution (of course), and the fix, hear me out...

  • Go to Terraform Cloud -> <workspace> -> Settings -> General
  • Change Execution Mode to Remote
  • This makes Terraform Version visible...
  • Change it to match your local version
  • Change Execution Mode back to Local
  • Hit Save Settings
  • Fixed! 🙄

@qwerty1979bg
Copy link

@mt-inside ,
This particular bug was fixed by #27498 in Terraform v0.14.5

@lado936
Copy link

lado936 commented Feb 10, 2021

@qwerty1979bg it still happening to me,

Failed to load state: state snapshot was created by Terraform v0.14.6, which is newer than current v0.13.6; upgrade to Terraform v0.14.6 or greater to work with this state

@qwerty1979bg
Copy link

@lado936 I suspect that your case is not the same as the one described by mt-inside.

This thread has been running for a while and people have mixed a few separate issues.

If you are using Terraform Cloud, your best bet is to open a case with support, so they can help you with your particular error / issue:
https://support.hashicorp.com/hc/en-us

If you are using Terraform CLI, you can start a thread in the community:
https://www.terraform.io/community.html

If you are sure this is actually related to the original poster's issue, please provide additional details.

@lado936
Copy link

lado936 commented Feb 10, 2021

@qwerty1979bg its definitely Terraform CLI, i am referencing other module inside terraform which is in fact ran by terraform 0.14.6, i am using 0.13.6 and have the error showed up, as referenced module is ran by 0.14.6 i cant use it inside my module.

@simov
Copy link

simov commented Feb 18, 2021

Not sure if this is even a bug in my case, but I was also stuck with a shared remote state on 0.14, where my project was still on 0.12 and I had to migrate it to 0.13 first.

As usual I did a plan using 0.13 locally, but before that instead of patching the remote state directly (on S3), I decided to pull it locally and patch my shared state source to use it as a local backend instead:

data "terraform_remote_state" "shared" {
  backend = "local"
  config = {
    path = "/home/.../${var.environment}.terraform.json"
  }
}

This is just a variation of some of the previous comments, but I found it safer to do this operation locally instead of patching the actual remote state that might as well be used by someone else in that moment.

@idavydiak
Copy link

I have found working solution for Terraform Enterprise here: https://support.hashicorp.com/hc/en-us/articles/360041299873

Error was: state snapshot was created by Terraform v0.13.6, which is newer than current v0.12.29; upgrade to Terraform v0.13.6 or greater to work with this state

@figaw
Copy link

figaw commented Mar 2, 2021

+1 for adding a -upgrade flag to terraform init to explicitly upgrade the version of the state, or fail with a warning similar to

Error: state snapshot was created by Terraform v0.12.29, which is older than current v0.13.5; use v0.12.29 or use `-upgrade` to upgrade to Terraform v0.13.5 or greater to work with this state

@yegorski
Copy link

yegorski commented Mar 2, 2021

Just hit this when going form 0.11.14 to 0.12.29. The error was

Error: state snapshot was created by Terraform v0.13.6, which is newer than current v0.12.29; upgrade to Terraform v0.13.6 or greater to work with this state

The issue was that the repo I was upgrading pulled in a 0.13.6 terraform_remote_state and that's what the error was really about.

@mikemartin1090
Copy link

@yegorski Thank you for pointing that out. My 12.25 workspace is pulling terraform_remote_state from a 12.26 workspace. What did you end up doing to resolve the issue?

@yegorski
Copy link

I removed data "terraform_remote_state" block that was pulling TF state of a projects that's on a version higher than 0.12. This was a big project that was pulling in and referencing state from multiple other projects.

@jslusher
Copy link

I just ran into this on an upgrade from 0.12.16 to 0.13.6 using s3 as my remote state storage. I have a lot of RDS instances and I got this error and terraform failed midway through the apply as a result. I had planned on backing out of the upgrade and so I tried a plan with my previous version and I ran into this issue. I then decided to try this method and edit the tf version in the .tfstate file and although the error went away it also now doesn't see anything in the state and it wants to create everything. At this point I'm just going to push forward since I don't want to know what happens if I try to apply that plan.

Plan: 53 to add, 0 to change, 0 to destroy.

@Joeltrane
Copy link

Joeltrane commented Jul 18, 2021

Has anyone tried using -lock=false to their terraform apply and/or terraform destroy commands?

I had a similar issue going from version 0.13.5 to 1.0.2. Including -lock=false allowed me to at least gracefully destroy everything and start over, but that was what I wanted to do. It might work on an apply ..

@Vincenius
Copy link

The solution for me was to download a previous version of the state from s3 and re-upload it to replace the one with the wrong version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests