Wrong default tool when both Terraform and OpenTofu are installed?

[ruijarimba]

Describe the bug

Terragrunt seems to use OpenTofu by default when both Terraform and OpenTofu are installed side-by-side.

Steps To Reproduce

1. Install both OpenTofu and Terraform tools into /usr/local/bin/ directory:

> ls -la /usr/local/bin/

# ..................
#40 0.714 -rwxr-xr-x 1 root root 88207512 May 22 15:51 terraform
#40 0.714 -rwxr-xr-x 1 root root 77416351 May 31 15:12 terragrunt
#40 0.714 -rwxr-xr-x 1 root root 82243584 May  8 15:35 tofu

2. Run any terragrunt command such as terragrunt init or terragrunt plan using configuration that doesn't use TERRAGRUNT_TFPATH nor --terragrunt-tfpath, as mentioned in the Expected behaviour.

Expected behavior

As mentioned in v0.52.0 , Terragrunt should use Terraform instead of OpenTofu unless applying one of:

• Option 1: Remove terraform binary from PATH
• Option 2: Define env variable TERRAGRUNT_TFPATH=tofu
• Option 3: When launching terragrunt, specify --terragrunt-tfpath tofu

Actual configuration:

{
  "TerraformBinary": "tofu",
  "TerraformCommand": "terragrunt-info",
  ...........
}

Expected configuration:

{
  "TerraformBinary": "terraform",
  "TerraformCommand": "terragrunt-info",
  ..........
}

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: v0.58.12
• OpenTofu version: v1.7.1
• Terraform version: v1.8.4

Environment details:

I am installing Terragrunt, OpenTofu and Terraform side-by-side in a Docker image

FROM mcr.microsoft.com/powershell:7.4-ubuntu-jammy

# ..... omitted for brevity

# https://github.com/hashicorp/terraform/releases
ENV TERRAFORM_VERSION 1.8.4

# https://github.com/gruntwork-io/terragrunt/releases
ENV TERRAGRUNT_VERSION 0.58.12

# https://github.com/opentofu/opentofu/releases
ENV OPENTOFU_VERSION 1.7.1

# ..... omitted for brevity

RUN \
    echo "Terraform v${TERRAFORM_VERSION} installation" \
    && curl --fail -Lo terraform.zip \
    "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" \
    && unzip terraform.zip -d /usr/local/bin/ \
    && rm terraform.zip \
    && terraform --version

RUN \
    echo "Terragrunt v${TERRAGRUNT_VERSION} installation" \
    && curl --fail -Lo /usr/local/bin/terragrunt \
    "https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64" \
    && chmod +x /usr/local/bin/terragrunt \
    && terragrunt --version

RUN \
    echo "OpenTofu v${OPENTOFU_VERSION} installation" \
    && curl --fail -Lo tofu.zip \
    "https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip" \
    && unzip tofu.zip -d /usr/local/bin/ \
    && rm tofu.zip \
    && tofu --version

Additional context

Add any other context about the problem here.

[yhakbar]

Sorry about the confusion!

We failed to update our documentation adequately, but this is the expected default behavior of Terragrunt.

If both Terraform and OpenTofu are installed on a host, Terragrunt will default to using OpenTofu unless specified otherwise.

[ruijarimba]

I'm not happy with this change. I do appreciate the effort in supporting and somehow motivating the use of OpenTofu, but making it the default tool is a BREAKING CHANGE.

How many issues will you cause to people still using Terraform that will upgrade Terragrunt just to find out their pipelines etc are broken because suddenly the default tool changed?

[weisdd]

@yhakbar The annoying side-effect of the change is that opentofu updates provider URLs in tfstate making them point at opentofu registry, and some of the providers in their registry are signed with keys that cannot be validated by terraform (e.g. vancluever/acme v2.22.0).
And it's not enough to just enforce terraform binary through a flag/terragrunt.hcl, you also have to manually modify tfstate to make it point at hashicorp registry. Without that, the init phase would fail due to uknown provider signature.
So I would agree with @ruijarimba, it's a breaking change, and it shouldn't have happened in the first place.

[MatthewJohn]

I'm not happy with this change. I do appreciate the effort in supporting and somehow motivating the use of OpenTofu, but making it the default tool is a BREAKING CHANGE.

How many issues will you cause to people still using Terraform that will upgrade Terragrunt just to find out their pipelines etc are broken because suddenly the default tool changed?

This also becomes really difficult because (assuming semantic versioning, though this may not be being followed), since the releases are pre-1.0.0, the versioning becomes really difficult to indicate breaking changes.

I completely agree that this feels like a breaking change - if a user starts with the expectations that they are using Terraform and suddenly switch to OpenTofu, it feels dangerous. Whist Terraform and OpenTofu were compatible, they even state themselves that they are only compatible up to 1.6 ^1

This means that the 1.7 releases of Terraform/OpenTofu can (and do) have feature divergence. We can see this in the OpenTofu state encryption and the Terraform features implemented in OpenTofu that sparked controversy (and how this may/may not affect future implementation of like-for-like features).

I'm not arguing either way that people should use one or the other, nor that one or the other is better (in any means of the word). However, what I am saying is that treating them as the same product and are interchangeable in a patch release is dangerous.

^1 - https://opentofu.org/ - "What are the differences between OpenTofu and Terraform?":

On the technical level, OpenTofu 1.6.x is very similar feature-wise to Terraform 1.6.x. In the future, the projects feature sets will diverge.

The other main difference is that OpenTofu is open-source, and it's goal is to be driven in a collaborative way with no single company being able to dictate the roadmap.

Edit: To be clear, this concern doesn't come from whether, after upgrading, the a pipeline fails or such, it's more that users won't realise, they go 6 months or a year down the line and then they realise they try to use a feature that doesn't work (because it's not in the product that they're actually using), they attempt to switch and the state is incompatible and they end up with a mess of a migration.

I'll also definitely end this on: this is still a pre-1.0.0 release and users should pay attention to this fact - so, I guess a plea to terragrunt to release a major version and follow semantic versioning and to users to not ignore the fact that something is pre 1.x.x

[josh-padnick]

Gruntwork CEO here. First, I'd like to apologize. After reading through your comments, I believe the root issue here is that we made a breaking change but released it as a patch change, leading to a surprise that in fact there was a breaking behavior for some users. This issue proved to be helpful feedback for us, and we created #3178 to address the root cause.

One of our guiding mantras for how we build Terragrunt is "do what's best for the Terragrunt community," and this was a situation where we could have done better.

As a resolution, we plan to issue a breaking release where the release notes reference the original release that contained switching the default to OpenTofu. This way, all users who study breaking change releases will catch that we made this change earlier.

Thanks again for the feedback, and any other suggestions you have either here or on #3178 are welcome!

[MatthewJohn]

Thanks @josh-padnick,

Not sure which comment this was directly directly geared at, but from my PoV, that's exactly it, so thank you for this and for creating the issue :)

Matt

[yhakbar]

Note that the following releases have been retroactively updated into include notices to help users understand the change in default tool for Terragrunt:

• https://github.com/gruntwork-io/terragrunt/releases/tag/v0.57.12
• https://github.com/gruntwork-io/terragrunt/releases/tag/v0.58.0

We're hoping that the updated notes on the releases will make it clear that additional action has to be taken to make the upgrade to a version > v0.57.12 if users are currently using Terraform.

[bradiosd]

@yhakbar The annoying side-effect of the change is that opentofu updates provider URLs in tfstate making them point at opentofu registry, and some of the providers in their registry are signed with keys that cannot be validated by terraform (e.g. vancluever/acme v2.22.0). And it's not enough to just enforce terraform binary through a flag/terragrunt.hcl, you also have to manually modify tfstate to make it point at hashicorp registry. Without that, the init phase would fail due to uknown provider signature. So I would agree with @ruijarimba, it's a breaking change, and it shouldn't have happened in the first place.

I had this exact issue today. I had to step into very dangerous territory requiring me to delete the state files and re-push them. I have explained my steps to fix in this issue thread.

alpine-docker/terragrunt#36 (comment)

[kevcube]

I was also caught by this problem, and I assumed it was a bug because when I checked the PR adding tofu support it mentions that default behavior will be to use terraform, and fallback to tofu if terraform is not found. https://github.com/gruntwork-io/terragrunt/releases/tag/v0.52.0

[yhakbar]

Thanks for the call out, @kevcube .

I've updated the release notes with a note that the behavior changed in the future release of 0.57.12.