cdk bootstrap: Error running cdk bootstrap with custom template

[saiumesh-appfire]

Describe the bug

We’re encountering an inconsistent issue with CDK bootstrap when using a custom template. Here’s the scenario:

1. We initially bootstrap using the command:

cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml

This works as expected, and the resources are created successfully.

2. However, when we update the template to include new permissions and attempt to reapply it with:

cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml --force or cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml

(with or without the --force flag),the command fails intermittently with below error

Environment aws://<accountId>/us-east-1 failed bootstrapping: Error: Trying to perform an operation that requires a bootstrap stack; you should not see this error, this is a bug in the CDK CLI.
    at NoBootstrapStackEnvironmentResources.lookupToolkit (/Users/apple/.volta/tools/image/packages/cdk/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:614:6763)
    at makeBodyParameter (/Users/apple/.volta/tools/image/packages/cdk/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:603:14404)
    at deployStack (/Users/apple/.volta/tools/image/packages/cdk/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:613:283)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _BootstrapStack.update (/Users/apple/.volta/tools/image/packages/cdk/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:614:10772)
    at async /Users/apple/.volta/tools/image/packages/cdk/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:821:1671
Trying to perform an operation that requires a bootstrap stack; you should not see this error, this is a bug in the CDK CLI.

3. Interestingly, if I manually update the same template through the AWS CloudFormation console, it works without any issues.

Is there something we might be missing, or could this be a bug in the CDK bootstrap process? We’d appreciate your guidance on troubleshooting this.

Thank you!

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.171.1 (build a95560c)

Expected Behavior

• The updated template with new permissions should apply successfully when using the cdk bootstrap command.
• The --force flag should reliably overwrite the existing resources with the updated template, without requiring manual intervention through the AWS console.
• There should be consistent behavior between applying changes via cdk bootstrap and updating the CloudFormation stack manually in the AWS console.

Current Behavior

• The initial cdk bootstrap command with the custom template works as expected, creating the necessary resources.
• When updating the template with new permissions and reapplying it using cdk bootstrap (with or without the --force flag), the process fails intermittently
• The same template works perfectly fine when updated manually via the AWS CloudFormation console.

Reproduction Steps

1. Create a custom CloudFormation template (e.g., bootstrap-template.yaml) with specific permissions and resources for CDK bootstrap.
2. Run the initial bootstrap command: cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml
3. Modify the template to include new permissions or resources.
4. cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml --force or cdk bootstrap aws://<accountId>/us-east-1 --template bootstrap-template.yaml
5. Observe that the update fails intermittently
6. Manually update the same template through the AWS CloudFormation console.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.171.1 (build a95560c)

Framework Version

No response

Node.js Version

18.16.0

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

@saiumesh-appfire Good morning. Somehow, I'm unable to reproduce the issue using CDK version 2.171.1 (build a95560c). Used below steps:

• Exported existing bootstrapped template in region us-east-2 using command cdk bootstrap --show-template > bootstrap-template-us-east-2.yaml.
• Added the below bucket resource (for testing purposes) in the exported bootstrapped template:

    TestStagingBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName:
          Fn::If:
            - HasCustomFileAssetsBucketName
            - Fn::Sub: ${FileAssetsBucketName}-test-issue32330
            - Fn::Sub: cdk-${Qualifier}-assets-${AWS::AccountId}-${AWS::Region}-test-issue32330
        AccessControl: Private

• Executed command cdk bootstrap aws://<<ACCOUNT-ID>>/us-east-2 --template bootstrap-template-us-east-2.yaml --force. The command executed successfully:

  Using bootstrapping template from bootstrap-template-us-east-2.yaml
   ⏳  Bootstrapping environment aws://<<ACCOUNT-ID>>/us-east-2...
  Trusted accounts for deployment: (none)
  Trusted accounts for lookup: (none)
  Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
  CDKToolkit: creating CloudFormation changeset...
   ✅  Environment aws://<<ACCOUNT-ID>>/us-east-2 bootstrapped.
  

Kindly note have the default credentials configured on my system has AdministratorAccess.

Please share the following:

• What is the Last Known Working CDK Version. You mentioned 2.171.1 (build a95560c), but also reported the issue in the same version.
• What changes are you making in bootstrap template? Could you please share the minimal changes that reproduces the issue?
• Do you have sufficient permissions to deploy the changes (new resource/permissions) you are making to bootstrap template?
• Could you execute your cdk bootstrap --force command with --verbose and --trace flags?
  Just FYI, I see warning/message when I use --verbose --trace flags, but the bootstrap process is successful:

  ...
  [09:00:55] Reading cached notices from /Users/<<USERNAME>>/.cdk/cache/notices.json
  [09:00:55] Toolkit stack: CDKToolkit
  Using bootstrapping template from bootstrap-template-us-east-2.yaml
   ⏳  Bootstrapping environment aws://<<ACCOUNT-ID>>/us-east-2...
  [09:00:55] Resolving default credentials
  [09:00:57] Looking up default account ID from STS
  [09:00:57] <<ACCOUNT-ID>> arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/Admin/<<REDACTED>>
  [09:00:57] Default account ID: <<ACCOUNT-ID>>
  [09:00:58] Waiting for stack CDKToolkit to finish creating or updating...
  [09:00:58] Retrieved account ID <<ACCOUNT-ID>> from disk cache
  Trusted accounts for deployment: (none)
  Trusted accounts for lookup: (none)
  Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
  [09:00:58] CDKToolkit: checking if we can skip deploy
  [09:00:58] CDKToolkit: forced deployment
  [09:00:58] CDKToolkit: deploying...
  [09:00:58] Could not determine the bootstrap stack name: Error: Trying to perform an operation that requires a bootstrap stack; you should not see this error, this is a bug in the CDK CLI.
  [09:00:59] Removing existing change set with name cdk-deploy-change-set if it exists
  [09:00:59] Attempting to create ChangeSet with name cdk-deploy-change-set to update stack CDKToolkit
  CDKToolkit: creating CloudFormation changeset...
  [09:01:00] Initiated creation of changeset: arn:aws:cloudformation:us-east-2:<<ACCOUNT-ID>>:changeSet/cdk-deploy-change-set/95168131-b082-4c20-bea8-6c1fcee36e95; waiting for it to finish creating...
  [09:01:00] Waiting for changeset cdk-deploy-change-set on stack CDKToolkit to finish creating...
  [09:01:00] Changeset cdk-deploy-change-set on stack CDKToolkit is still creating
  [09:01:06] Initiating execution of changeset arn:aws:cloudformation:us-east-2:<<ACCOUNT-ID>>:changeSet/cdk-deploy-change-set/95168131-b082-4c20-bea8-6c1fcee36e95 on stack CDKToolkit
  [09:01:07] Execution of changeset arn:aws:cloudformation:us-east-2:<<ACCOUNT-ID>>:changeSet/cdk-deploy-change-set/95168131-b082-4c20-bea8-6c1fcee36e95 on stack CDKToolkit has started; waiting for the update to complete...
  [09:01:07] Waiting for stack CDKToolkit to finish creating or updating...
  [09:01:07] Stack CDKToolkit has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS (User Initiated))
  CDKToolkit | 0/3 | 9:01:06 AM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
  CDKToolkit | 0/3 | 9:01:09 AM | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | TestStagingBucket 
  CDKToolkit | 0/3 | 9:01:10 AM | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | TestStagingBucket Resource creation Initiated
  [09:01:13] Stack CDKToolkit has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
  [09:01:18] Stack CDKToolkit has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
  [09:01:23] Stack CDKToolkit has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
  CDKToolkit | 1/3 | 9:01:22 AM | CREATE_COMPLETE      | AWS::S3::Bucket         | TestStagingBucket 
  CDKToolkit | 2/3 | 9:01:24 AM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | CDKToolkit 
  CDKToolkit | 3/3 | 9:01:24 AM | UPDATE_COMPLETE      | AWS::CloudFormation::Stack | CDKToolkit 
  [09:01:30] Stack CDKToolkit has completed updating
   ✅  Environment aws://<<ACCOUNT-ID>>/us-east-2 bootstrapped.
  [09:01:30] Reading cached notices from /Users/<<USERNAME>>/.cdk/cache/notices.json
  

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[saiumesh-appfire]

Hi @ashishdhingra ,Thank you for your response!

This issue seems to occur intermittently, not every time, which makes it a bit tricky to debug. Unfortunately, I’m unable to share the internal template we’re using. Given that the error message states, you should not see this error, this is a bug in the CDK CLI, do you have any suggestions on how to proceed if this happens again?

Thanks for your help!

[ashishdhingra]

Hi @ashishdhingra ,Thank you for your response!

This issue seems to occur intermittently, not every time, which makes it a bit tricky to debug. Unfortunately, I’m unable to share the internal template we’re using. Given that the error message states, you should not see this error, this is a bug in the CDK CLI, do you have any suggestions on how to proceed if this happens again?

Thanks for your help!

@saiumesh-appfire You may try using --verbose --trace flags when this issue occurs to capture the trace, thereafter opening new issue with the findings.

We would close this issue for now.

Thanks,
Ashish

[saiumesh-appfire]

Thanks, @ashishdhingra I'll close the issue now and will open it with traces once we have this error. Thanks for your help!!

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.