feat(lib): cross-stack references

[DanielMSchmidt]

Closes #651

General

This PR adds support for cross stack references (initially for the local and Terraform Cloud backend being used in the source stack).
A Cross Stack Reference is a reference that is used in a different stack than it's created, e.g.:

export class SourceStack extends TerraformStack {
  public diceRoll: Integer;
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new RandomProvider(this, "random");

    this.diceRoll =  new Integer(this, "int", {
      min: 0,
      max: 6,
    });
  }
}

export class ConsumerStack extends TerraformStack {
  constructor(
    scope: Construct,
    id: string,
    inputs: { randomValue: number }
  ) {
    super(scope, id);

    new LocalProvider(this, "local");

    new File(this, "dice-roll-result", {
        filename: "dice-roll-result.txt",
        content: String(inputs.randomValue)
    })
  }
}

const app = new App();
const src = new SourceStack(app, "source");
new ConsumerStack(app, "destination", {
  randomValue: src.diceRoll.result, // we use the reference from the source stack here
});
app.synth();

How to review

I'd recommend going commmit by commit, they should all make sense by themselves

[jsteinich]

Does this supersede #1179? There were some comments there that may still apply here.

[DanielMSchmidt]

Yes it will, I left the other PR open so that I don't forget to look through the comments

[ansgarm]

Just came across this when reading a former proposal for cross stack references I did a while back: aws/aws-cdk#12778
We should at least create a follow-up task to check whether this could happen to us as well.

[DanielMSchmidt]

@skorfmann does this test case cover your concern around complex computed lists or did I do sth wrong here? 103a5be

[skorfmann]

What happens if I have deployed a producing and consuming stack. After the deployment, I remove the reference and hence the automatically created output. I have a hunch that I might run into problems planning / deploying the consuming stack once the output of the producing stack is gone.

That's a great point, @ansgarm brought that up as well, this is sth the AWS CDK struggles with. I created a test to check and it does not cause problems for TF: https://github.com/hashicorp/terraform-cdk/pull/1416/files#diff-99c6047d4d933587477aa2e3622ea8bc6fa07034cc9b32861c3f3c07d74ba734R81 and here is the main.ts for that test: https://github.com/hashicorp/terraform-cdk/pull/1416/files#diff-5162569d3dc22ac1cf854a3aaaf4b4cc41eb9a7ece4f578006ce94b6fcb321ceR140

The links appear to be broken. But anyway, if it works that's ok.

[skorfmann]

The output of the producing stack should contain the fully resolved value of Terraform. That a function is involved is not a concern of the consuming stack.

Is that more of a general guideline or would you say that's important in the first iteration? We could do a follow-up issue for this, would that work for you?

I think it would be good to have this properly working from the beginning, seems very unexpected to me.

[skorfmann]

@skorfmann does this test case cover your concern around complex computed lists or did I do sth wrong here? 103a5be

It's not really straightforward to understand what's actually being tested there. Here's my case: I was passing the entire domain validation options object (ComplexComputedList), not a single element. Not even sure if that's a thing which should be working or which should be caught with an error.

[skorfmann]

I wonder if this is a real use-case though, wouldn't folks use the attributes of a resource? I think this is never a good idea and would argue for disallowing passing these into outputs and terraform functions in general

Passing complete resources into an output is definitely valid. Terraform accepts that and there was a ticket fairly recently where a user requested the functionality and we made it work. It's not possible to pass to any other resource though and also seems a bit far fetched to pass into another stack's output.

So, this should raise an error when referencing across stacks then, correct?

[skorfmann]

Haven't gone through the code since others did this already. As laid out in the comments I'd add a few more explicit errors for things which are not working (yet). Plus, the functions magically appearing in a referencing stack should be fixed. Other than that, I think it's good and works as expected 👍

[DanielMSchmidt]

I'm going to create follow up issues for the outstanding work and mention them here 👍

[DanielMSchmidt]

• Fn should be saved in origin stack: Cross Stack References using Functions should save the functions output in origin stack #1480
• Passing complex objects / resources across stacks: Cross Stack References should support or error on passing entire objects #1481

[dhmistry3]

Is there an example of how to do this in python?

[DanielMSchmidt]

We have a test case in Python, hope that helps :) https://github.com/hashicorp/terraform-cdk/blob/main/test/python/cross-stack-references/main.py

The usage is pretty similar to Typescript, passing a value created in one stack into another one does the trick

[dhmistry3]

@DanielMSchmidt Thanks that is helpful. However, I am trying to pass an aws ARN from one stack to another. When i set it in one stack and reference it in another, the variable gets set as

${module.s3_bucket_id.s3_bucket_arn}

and then i get an error of

│ No module call named "s3_bucket_id" is declared in the root module.

[DanielMSchmidt]

@dhmistry3 Could you create an issue for this with a reproducible example so that I can take a look?

[dhmistry3]

@DanielMSchmidt So it worked in the simplified version i made (https://github.com/dhmistry3/cdktf-cross-stack) so there must be something wrong i am doing in my main code. Thanks, i'll ask if i have any more questions.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.