(core): cdk synth always synthesizes every stack

[Townsheriff]

❓ General Issue

I have multiple stacks in my project and want I deploy only one of them all other stack assets must be present in file tree or else I get ENOENT: no such file or directory, stat....

I did a little digging and figured:
a. cdk cli spawns process ts-node /bin/script.ts
b. cdk cli does not pass stack name to aws-cdk, therefore I cannot know which should be synthesized
c. aws-cdk generates all cloudformation jsons for all stacks it can find and cdk cli picks the ones it needs

I think it would be smart to provide the name of stack and generate only provided one, it would be faster and would not require all assets to be present.

PS.
With aws-cli I'm referening to aws-cdk installed globally and with aws-cdk I refer to project dependency.

The Question

1. I wonder what was the reasoning for current approach and should it be changed?
2. Maybe I'm missing something and this can be easily resolved?

Environment

• **CDK CLI Version: 1.27.0 (build a98c0b3)
• **Module Version: "@aws-cdk/core": "^1.27.0"
• **OS: MacOs Mojave
• **Language: Typescript

Other information

[shivlaks]

@Townsheriff Did you use the --exclusively flag when synthesizing and deploying the stack that you wanted to work with?

I'm working on a repro so I can dive deeper into the behaviour. I'm going to create an app with 2 stacks (both leveraging assets) and then try to synth and deploy one of them and also providing the --exclusively flag. Let me know if there's anything else I'm missing!

[Townsheriff]

@shivlaks I did not use the option, perhaps that is solution. I ended up adding env variable which I would use in ts file to determine which stack should be synthesised.

[jgondron]

@shivlaks Were you able to reproduce with the --exclusively flag in an isolated app? I'm pretty sure I'm experiencing this exact issue even while using the flag, although in a more complicated environment.

[shivlaks]

@jgondron apologies, I have not had a chance to follow up on this one.

Can you describe your environment/setup at a high level?
It would be helpful to continue collecting information for anyone who is able to pick this issue up before me.

[jgondron]

@shivlaks I have a multistack app with stackA and stackB with corresponding pipelines for deploying each. stackA has things like const codeAsset = lambda.Code.fromAsset(props.lambdaCodePath). The pipeline for stackB needs to exclusively deploy stackB with something like cdk deploy stackB --require-approval never --exclusively, but since the pipeline for stackB does not have the app code for stackA it’s throwing errors such as ENOENT: no such file or directory, stat '/<some codebuild path>/appA/src'

Our current work around is to add something like this to stackA before the lambda.Code.fromAsset call:

if(!fs.existsSync(props.lambdaCodePath)){
  this.node.addError(`Cannot deploy this stack. Asset path not found ${props.lambdaCodePath}`);
  return;
}

Doing this protects it from throwing an exception, but still fails to deploy stackA if it's app source is not found. Perhaps lambda.Code.fromAsset could be changed to perform this type of error handling itself?

[dansalias]

I am encountering the same issue.

Synthesising or attempting to deploy a stack with the --exclusively flag when another stack in the app references a nonexistent path with lambda.Code.fromAsset('/path/that/does-not/exist') fails with the message Cannot find asset at /path/that/does-not/exist.

I've put a minimal reproduction together at https://github.com/dansalias/tmp-cdk-6743

[dansalias]

In my case I have a repository with a number of services, each with their own build process (to compile lambdas) and CDK stack, managed through the same CDK app. When a given service is updated its build process is triggered, followed by deployment. The way the cdk is currently setup requires all services to be built even if just a single service is to be deployed.

I'm happy to work on a PR for this but it would be useful to have some guidance. I imagine it would be a case of advancing this logic so that unspecified stacks aren't even synthesised in the first place.

[dansalias]

@ericzbeard @rix0rrr are we able to revisit this please?

[diesal11]

Any update on this?

[danieldspx]

@shivlaks Any update on this? I think that be forced to build everything just to deploy one stack is really bad.

[danieldspx]

What I've done so far, for those looking for a workaround, is to rely on bundlingRequired property in the stack. It is similar to this:

if (!this.bundlingRequired) {
    // We must skip undesired stacks to be able to deploy specific stacks.
    // Refer to: https://github.com/aws/aws-cdk/issues/6743
    console.info('Skipping ' + this.stackName);
    return;
}

This works because in case your stack is selected to be deployed bundlingRequired will be true, and in any other case it will be false. Note that it also works for the bootstrap command gracefully. I did not include this piece of code in stacks that dont use lambda.Code.fromAsset because there is no need for that.

[Ribosom]

I really like the possibility to simply deploy typescript lambdas with cdk. However, this issue is the only big pain I have with cdk. If I have to deploy one stack quickly (in a dev environment), I have to wait multiple times. We have mutiple step function with multiple lambdas each. Every step function is in one stack. If we only want to deploy one step function to test it, we always have to wait for all assets of all stacks to be bundled. If you made a mistake and want to deploy again, you have to wait again for everything to be bundled.

Is there any workarround? Is there a best practice to design the stacks so this is not an issue?

[dguisinger]

It seems like the proper solution is to just make separate CDK projects for each stack....or I guess switch back to Terraform. I can't believe this comment thread is still going with no resolution after 3 years. Its so frustrating, somehow AWS makes their active codebase on the tool they tell everyone to use feel like abandonware.

In my case I have two stacks:

1. Stack 1 creates the build pipeline
2. Stack 2 creates the deployment

The build pipeline builds the lambda functions, outputs zip files.
The deployment cdk is then supposed to run to deploy the zip files.

But since -e doesn't actually work, I can't deploy the pipeline stack without using one of the above hacks.

[guysqr]

I really like the possibility to simply deploy typescript lambdas with cdk. However, this issue is the only big pain I have with cdk. If I have to deploy one stack quickly (in a dev environment), I have to wait multiple times. We have mutiple step function with multiple lambdas each. Every step function is in one stack. If we only want to deploy one step function to test it, we always have to wait for all assets of all stacks to be bundled. If you made a mistake and want to deploy again, you have to wait again for everything to be bundled.

Is there any workarround? Is there a best practice to design the stacks so this is not an issue?

--hotswap

Hot swapping
Use the --hotswap flag with cdk deploy to attempt to update your AWS resources directly instead of generating an AWS CloudFormation change set and deploying it. Deployment falls back to AWS CloudFormation deployment if hot swapping is not possible.

https://docs.aws.amazon.com/cdk/v2/guide/cli.html

[comcalvi]

The suggestion here: #28136 (comment) is the recommended best practice to avoid this issue. I will get our developer guide updated to reflect this.

The core reason behind this recommendation is that cdk synth is executing your CDK application. Your CDK application has defined some stacks, perhaps like this:

import { StackA, StackB } from '../lib/temp-project-stack';

const app = new cdk.App();
new StackA(app, 'StackA', {});
new StackB(app, 'StackB', {});

new StackA(...) will instantiate the stack, which instantiates all constructs defined in StackA. Any constructs in StackA that use assets, like Lambda functions, will discover their assets during this time, and will throw errors if those assets aren't present.

The errors complaining about non-existent assets are thrown during the execution of your CDK program, so the only way to prevent them is to prevent the CDK from executing that code. The best way to do that is to use some environment variables to prevent the execution of that stack code, as shown by @tmokmss.

[Dzhuneyt]

The errors complaining about non-existent assets are thrown during the execution of your CDK program, so the only way to prevent them is to prevent the CDK from executing that code. The best way to do that is to use some environment variables to prevent the execution of that stack code, as shown by @tmokmss.

No, the best way would be for the CDK CLI to be smart enough to understand that new StackA() and its child Constructs will never be needed, if the CLI was invoked with any of these:

• cdk synth "StackB"
• cdk deploy "StackB"
• cdk deploy "*B"

---

Btw, the "cdk deploy" command already understands (somewhere at a later stage after synth) that it needs to only consider stacks that match the glob pattern provided as first argument. I don't see why is it so difficult to consider this glob pattern in the synth stage also, to conditionally skip redundant synths of stacks that don't match the glob pattern.

[Dzhuneyt]

Actually, I don't see why the use case is so difficult to understand.

The command speaks for itself. When an engineer runs cdk synth "StackB" - the intention is clear as day. I want to synthesize StackB. Period.

I don't want to synthesize the whole app and throw away all stacks, but StackB. If that was the case, a better signature for the command would have been cdk app synth --filter "StackB" or something.

[comcalvi]

I understand the use case; it makes sense that cdk synth StackA would only synthesize StackA.

The problem is that synthesis operates on the entire CDK App, and requires executing the command specified in the app key of cdk.json or --app. This command is usually something like ts-node bin/my-app.ts. The CDK starts a new process to synthesize your app. That process will just execute your app as you have written it, including new StackB() if it's present.

Synthesis, as a consequence of its current design, cannot be made to operate on individual stacks in the way you have described. I have thought of the following modifications to the design of synthesis that may support this use case, but I don't believe any of them solve more problems than they create.

As far as I am aware, there is no way to tell node (or all the other language runtimes we support) to selectively ignore constructor calls to classes that do not match a certain glob string.

We can't catch and ignore errors thrown from this process, because any errors in that process will terminate it. The best we can do is restart it, and that will just rethrow the same error. Maybe we can workaround this by loading and executing that code at runtime (not sure if this is really possible), but even if we do that, it's not really selectively synthesizing; it's just ignoring certain errors.

Maybe we can provide a custom node runtime that allows us to pass this globstring to the process and selectively ignore constructor calls, but this would need to be very carefully designed and maintenance tradeoffs would need to be considered; I don't expect it would be worth it.

Maybe we can modify the jsii compiler to inject special code into stack constructors to make it ignore certain stack calls, but it's not clear how viable or possible this is.

A less generalized solution, but one that might be more viable, is to vend different cdk init templates that create a new file for each stack, and then a separate file for the whole app. Then if you pass a stack name to cdk synth, it could figure out which file to run based off the glob string passed. This might work, but we'd have to think it through for every language.

I don't like any of these options, because they violate the core premise of synth does; it's an App-level operation. If you want to not synthesize certain stacks within that App, you should modify the source of that App to use the environment variables method described above.

Please let me know if there are implementation paths that I have missed. Contributions are always welcome.

[aws-sde]

Ultimately the reason I'd like to synthesise only one stack is to save time. More generally, it would be great if it worked like build systems, which are smart enough to only build what has changed. Why should I wait to resynthesise the whole app when my cdk.out directory has a bunch of perfectly good and up-to-date templates?

Let's say I have a dependency chain of stacks A → B → C (i.e. A depends on B, and B depends on C). Intuitively, when I run cdk synth B --exclusively --app cdk.out, A and B should be resynthesised and written to cdk.out. C should not be resynthesised (unless circular dependencies are a thing?) Or maybe --exclusively should really just do B, and have another mode that also synthesises stacks that depend on B.

@comcalvi For the sake of synthesising one stack, is it necessary to avoid construction of the other stack classes? Can it still construct the stacks, but when it comes to synthesis, re-use what's in cdk.out for stacks that are dependencies of the target stack? Or is it not possible to "reverse engineer" those templates in cdk.out back into constructs/nodes for synthesis?

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[rehanvdm]

This issue has been open for 4 years, it does not look like a core change would be made anytime soon. If you need to DIY it, you can do so without an extra context variable:

You can get the stack/bundle specifier and then conditionally (using if statements) create the stacks within your code. Example:

If I do this CDK command:

cdk diff "workloads-base-*" --exclusively

Then you can find the stack identifier in the environment variable below, it will print:

console.log(JSON.parse(process.env.CDK_CONTEXT_JSON || "{}")['aws:cdk:bundling-stacks']);
// Outputs: [ 'workloads-base-*' ]

This can then be used to create if conditions in your code to only create/instantiate certain stacks.

Is it "hacky" and ugly, yes. But it's the only way I know of how to do it atm.

[Dzhuneyt]

@comcalvi a relatively low-hanging fruit type of solution to this problem, that combines the suggestion from the latest comment by @rehanvdm, might be the following:

Since all Stacks that are part of a CDK app are forced to call super(scope, id, props); as the first line of their constructor, as part of the CDK design, this also means that CDK itself as a framework has the possibility to execute some preliminary logic in that parent constructor, before the "custom code" is executed in that constructor.

This means that this super constructor has the opportunity to "analyze" which stacks the user wants synthesized (by parsing the process.env.CDK_CONTEXT_JSON for example` or something similar).

In that way, that parent constructor can understand if the "current stack" needs synthesis or not, and if not - do an early return or signal to subsequent code that it can also skip doing heavy lifting (e.g. signal to the Lambda bundler to not build the Lambdas of this stack, etc).

Does this sound like a feasible solution? Any blockers?