cli: when cdk.out is missing, cannot synthesize or list

[allister-db]

I deleted my cdk.out/ directory hoping to freshly generate its contents next time I ran cdk synth, but instead I get a file not found error.

Before I deleted cdk.out, synth, ls, and deploy were working fine.

Reproduction Steps

1. rm -r cdk.out
2. cdk ls

Error Log

ENOENT: no such file or directory, open 'cdk.out/manifest.json'

Environment

• CLI Version : 1.19.0
• Framework Version: 1.16.0
• OS : MacOS Catalina
• Language : Python
• Node version: v12.14.0

Other

> cdk --verbose ls
CDK toolkit version: 1.19.0 (build 5597bbe)
Command line arguments: {
  _: [ 'ls' ],
  verbose: true,
  v: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  long: false,
  l: false,
  '$0': '/usr/local/bin/cdk'
}
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "python3 private_load_balancer.py"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 private_load_balancer.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to us-west-2
Resolving default credentials
Retrieved account ID X from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to X
context: {
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'us-west-2',
  CDK_DEFAULT_ACCOUNT: 'X',
  CDK_CONTEXT_JSON: '{"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.19.0'
}
ENOENT: no such file or directory, open 'cdk.out/manifest.json'
Error: ENOENT: no such file or directory, open 'cdk.out/manifest.json'
    at Object.openSync (fs.js:440:3)
    at Object.readFileSync (fs.js:342:35)
    at new CloudAssembly (/usr/local/lib/node_modules/aws-cdk/node_modules/@aws-cdk/cx-api/lib/cloud-assembly.ts:77:36)
    at Object.execProgram (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:85:10)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at AppStacks.synthesizeStacks (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/stacks.ts:205:24)
    at AppStacks.listStacks (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/stacks.ts:179:22)
    at AppStacks.selectStacks (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/stacks.ts:109:20)
    at cliList (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:337:20)
    at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:188:16)

This seems very similar to a previous issue: #3717 but I did not find a clear resolution in the comments on issue 3717. One comment said "went away when I updated my cdk.ts" but it is not clear to me whether I can or should do that. I installed CDK by npm install -g aws-cdk, so I would expect all TypeScript to be "inside the black box".

---

This is 🐛 Bug Report

[allister-db]

Adding an app.synth() call to my Python script made it start working.

Before:

def main():
  app = App()
  PrivateBalancerStack(app, MyEnv.name, env=AWS_ENV, description="Private ALB resources.")

> rm -rf cdk.out cdk.context.json ; cdk ls
ENOENT: no such file or directory, open 'cdk.out/manifest.json'

After:

def main():
  app = App()
  PrivateBalancerStack(app, MyEnv.name, env=AWS_ENV, description="Private ALB resources.")
  app.synth()

> rm -rf cdk.out cdk.context.json ; cdk ls
private-alb-stack

Perhaps I misunderstood the way synthesis works (I thought app.synth() would not be necessary if I am using cdk synth from the command line). If this is simply operator error, let me know and I will close this case.

[allister-db]

Worth noting: after the stuff in cdk.out/ is generated from a cdk synth using the Python code that includes app.synth(), if I remove the app.synth() call from my Python, stuff still looks like it is working, but nothing in cdk.out/ gets updated.

So maybe the issue is:

1. I did misunderstand, and app.synth() is crucial.
2. Seeing correct-looking output from cdk synth and an exit code of 0 fooled me into thinking it was still doing work.

The moral of the story for me: don't remove app.synth() without understanding what it is there for. 😄
The moral of the story for CDK: is it a bug that cdk synth appears successful even when the misguided removal of app.synth() is making it not do the work it should be doing?

[shivlaks]

@allister-db sorry I'm jumping on this a bit late, but it seems you've gone through the discovery and learning aspect of the functionality :).

Resolving this, but feel free to reopen if you still have any outstanding questions.

[elreydetoda]

So, I know I am messaging on an old thread, but I recently came into this exact same situation and was curious why this (app.synth()) was necessary?

I have a co-worker who uses typescript with aws-cdk and in his commands he doesn't have the cdk.synth() function, but he just runs cdk synth from the CLI instead...so, does that mean there is a discrepancy between the cdk languages? (python and typescript)

@allister-db, did you ever figure out why this was required?

[peterwoodworth]

I've independently found that app.synth() is required in Python. No idea why personally

[DmitriKonnovNN]

So, I know I am messaging on an old thread, but I recently came into this exact same situation and was curious why this (app.synth()) was necessary?

I have a co-worker who uses typescript with aws-cdk and in his commands he doesn't have the cdk.synth() function, but he just runs cdk synth from the CLI instead...so, does that mean there is a discrepancy between the cdk languages? (python and typescript)

@allister-db, did you ever figure out why this was required?

As the matter of fact, AWS-CDK is written in TypeScript and runs on Node.js. All other aws-cdks are built on top of it.

[allister-beamable]

@allister-db, did you ever figure out why this was required?

Sorry, I never did dig deeply enough to find the "why" of needing an explicit app.synth() call when using the Python version of CDK. The good news is, as long as one knows it is needed, it is easy to always include it! 🙂

[paulied67]

I have been unable to find an explanation as to why and it would be nice to know why. Maybe my googling isnt googling so for now it is the equivalent of DoEvents in VB.

[andrewoffenden]

I had this issue, and none of the answers solved it. It seems apparent that this error can have a variety of causes. Here was mine.

import { 
  App,
  Environment,
 } from 'aws-cdk-lib';
import { EcsEc2DeploymentTestStack } from '../lib/ecs-ec2-deployment-test-stack';


(async  () =>  {
  const env: Environment = { 
    account: process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEFAULT_REGION ,
  };
  
  const application = new App();
  
  const ecsEc2Service = new EcsEc2DeploymentTestStack(application, 'EcsEc2TestService', { env: env });

  application.synth();
});

Here is the functioning code:

import { 
  App,
  Environment,
 } from 'aws-cdk-lib';
import { EcsEc2DeploymentTestStack } from '../lib/ecs-ec2-deployment-test-stack';


(async  () =>  {
  const env: Environment = { 
    account: process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEFAULT_REGION ,
  };
  
  const application = new App();
  
  const ecsEc2Service = new EcsEc2DeploymentTestStack(application, 'EcsEc2TestService', { env: env });

  application.synth();
})();

I had forgotten to invoke the asynchronous wrapper.