tfe_workspace has possible self-referencial block -> pull remote_state_consumer_ids out into separate resource

[JacobLey]

Use-cases

My personal case is I have a tfe_workspace for every microservice+environment combination.
Workspaces are dependent on each other and across environments. An example section of my .tf file looks like this:

resource "tfe_workspace" "foo" {
  for_each                  = toset(["qa","production"])
  name                      = "${each.value}-foo"
  global_remote_state       = false
  terraform_version         = "1.0.0"
  remote_state_consumer_ids = toset(compact([tfe_workspace.bar[each.value].id]))
  # ...
}

resource "tfe_workspace" "bar" {
  for_each                  = toset(["qa","production"])
  name                      = "${each.value}-bar"
  global_remote_state       = false
  terraform_version         = "1.0.0"
  remote_state_consumer_ids = toset(compact([each.value == "production" ? tfe_workspace.bar["qa"].id : ""]))
}

Note in the first case, qa-foo and production-foo are dependent on qa-bar and production-bar respectively. This works as planned.

The second case, qa-bar has a cross-environment dependency on production-bar. This causes the plan to error. This error is triggered by production-bar being defined in the same top-level block as qa-bar. So production-bar cannot reference qa-bar in it's own block (even though they are separate resources).

(Some terminology clarification) The real world dependency is qa-bar depends on production-bar. In terraform production-bar must declare this dependency, so that makes production-bar dependent on qa-bar in this case

The actual error is that production-bar["qa"] cannot reference production-bar["qa], despite the variable not being used in that "execution" of the ternary.
(Name changed from real case)

Error: Self-referential block
# ...file location...
Configuration for tfe_workspace.bar["qa"] may not refer to itself.

Considering that. tfe_workspace can logically depend on other resources of the same type, it would be useful to have the option to declare this connection in a separate resource, so terraform can deterministically figure out dependencies.

Attempted Solutions

1. remote_state_consumer_ids is only necessary when global_remote_state=false. So I could just open all workspaces up to one-another. For general security/design reasons I would prefer to not do that in name of POLA.
2. Hard-code that edge case. In my case these workspaces already exist, so id can be known in advance by me and hard-coded. So the line would change to something like each.value == "production" ? "abc123" : "". This is my current work around.
3. Don't use for_each to group resources, and declare each separately. In hindsight this would be the best design for this solution, but I don't have that currently implemented. Migrating to this would mean manually updating each resource pragmatically.

Proposal

Create a new resource tfe_workspace_remote_consumers. It will take only two parameters of workspace_id and remote_state_consumer_ids.

This ensures any workspace can depend on any other workspace. For backwards compatibility and simplicity, the tfe_workspace remote_state_consumer_ids can continue to act as normal.

This pattern of multiple types of resources being able to manipulate the same attributes is used other places like:
aws_sqs_queue and aws_sqs_queue_policy https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy
aws_security_group and aws_security_group_rule https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule

Similar to those implementations, a note should be included that users should only pick one. Similarly the tfe_workspace remote_state_consumer_ids should expect its value to be possibly set elsewhere if not explicitly defined. Although lifecycle { ignore_changes: [remote_state_consumer_ids] } should do the trick.

Request

While this feature would solve my issue, it seems possible there are other solutions around this. If someone could help with those that would be great.

Two thoughts are:

1. Somehow use the self value. provisioner can access its own resource with the self value. I guess a feature request to terraform core would be granting some self equivalent to the resource itself. Except I don't really want the resource... I want the group one level higher, with the other resource keyed by environment.
2. Only generate the reference sometimes... My understanding of the actual error is that when each.value == "production" ? tfe_workspace.bar["qa"].id : "" is calculated, tfe_workspace.bar["qa"].id's value is calculated even though the value is not eventually referenced when each.value == "qa". If there was some way of not doing that and instead tell terraform "don't calculate this value until absolutely necessary" that could also potentially solve this issue. Again this is probably a feature request to terraform core instead.

[pksunkara]

This is quite annoying not to have. Especially if we are trying to follow the best practices recommended by HashiCorp.

It's funny that the company recommends a best practice that it does not support.

[ktella]

We have the same issue too. Need someone from Hashi to take care of this.

[certara-mchamberland]

+1, my use case is I have shared infra (an AWS VPC, an ECS cluster and some odd bits) that need to be referenced by each module deployed to it (ECS service and supporting assets). Both of these use repositories and workspaces managed using Terragrunt, but I don't want to have to edit the shared workspace's remote_state_consumer_ids and apply every time I make a new workspace for the service module, nor create a circular dependency in Terragrunt between my workspace modules. If it was its own resource, the service modules could register themselves to the shared workspace's state.

Alternatively, my particular use case could be resolved by allowing remote_state_consumer_ids to target projects, as opposed to individual workspaces, as all the service modules co-exist in the same project as the shared module. Both solutions allow me to limit scope to only the workspaces I want.

[Satak]

I was just about to create the issue for this. This is clearly an oversight by HashiCorp and they should indeed create a separate resource item for this. Basically it is impossible to create workspaces with for_each and reference them as consumer ids because of the self-reference nature.

And what really bothers me is that this issue seems to be from 2021, it has the most up votes and it has not been fixed...

resource "tfe_workspace_remote_consumer" "remote_consumer" {
  workspace_id             = <workspace id>
  remote_state_consumer_id = <remote_state_consumer_id>
}

[brandonc]

@Satak Unfortunately this is a recurring issue with tfe_workspace... previously with workspace -> agent pool -> allowed workspaces dependency cycles. Luckily, I recently introduced tfe_workspace_settings to resolve this kind of loop and I added these attributes to that resource in #1524. So sorry it took so long to re-enter my radar but I thank you for your comment.

[Satak]

@Satak Unfortunately this is a recurring issue with tfe_workspace... previously with workspace -> agent pool -> allowed workspaces dependency cycles. Luckily, I recently introduced tfe_workspace_settings to resolve this kind of loop and I added these attributes to that resource in #1524. So sorry it took so long to re-enter my radar but I thank you for your comment.

Fantastic news, thanks so much!