Changing ssm parameter from string -> securestring throws me an cdk deployment error

[maddyexplore]

Describe the bug

I was trying to secure all strings in ssm parameter group and I found the cdk-bootstrap version there and I changed it into securestring, after that any deployment with cdk fails with error

AutomationStack: SSM parameter /cdk-bootstrap/<blabla>/version not a number

Expected Behavior

it should sense and decrypt with the km kwy

Current Behavior

throws me error

Reproduction Steps

secure all strings in ssm parameter group and I found the cdk-bootstrap version there and I changed it into securestring, after that any deployment with cdk fails

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.124.0

Framework Version

No response

Node.js Version

v21.6.1

OS

Windows

Language

Python

Language Version

python 3.10

Other information

No response

[ashishdhingra]

Bootstrap version SSM parameter is defined as DEFAULT_BOOTSTRAP_STACK_VERSION_SSM_PARAMETER here. During synthesize() operation, CheckBootstrapVersion rule would be added in the generated CloudFormation template along with the CfnParameter named BootstrapVersion as shown in example below:

Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
...
...
Rules:
  CheckBootstrapVersion:
    Assertions:
      - Assert:
          Fn::Not:
            - Fn::Contains:
                - - "1"
                  - "2"
                  - "3"
                  - "4"
                  - "5"
                - Ref: BootstrapVersion
        AssertDescription: CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI.

As per description Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. of BootstrapVersion parameter in the generated CloudFormation template, the value for this parameter is automatically retrieved from SSM Parameter Store. The CDK deployment is essentially a CloudFormation deployment and is handled by the CloudFormation service.

Per CDK Bootstrapping documentation,

• AWS CDK's built-in stack synthesizers is called DefaultStackSynthesizer (refer Stack synthesizers)
• During initialization of a stack, it is possible to specify a synthesizer as per below example in the above documentation:

  new MyStack(this, 'MyStack', {
    // stack properties
    synthesizer: new DefaultStackSynthesizer({
      // synthesizer properties
    }),
  });

• Here it is possible to override certain properties (as mentioned in Customizing synthesis section)
  • The property generateBootstrapVersionRule could be set to false and as per logic here, it should not emit BootstrapVersion CfnParameter and CheckBootstrapVersion CfnRule (these are added here).
    Unsure if this should be used since bootstrap version should always be validated.
  • Per Roles section at The bootstrapping template contract, the DefaultStackSynthesizer requires five IAM roles for five different purposes. For your use case, since the issue happens during deployment process, in your AWS environment, you could add necessary IAM permissions to the cloudFormationExecutionRole, which per above documentation is ARN of the role passed to CloudFormation to execute the deployments. and has format like arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region} (here the default value for ${Qualifier} is hnb659fds) deployment role created by CDK bootstrapping process. You could refer IAM permissions for using AWS default keys and customer managed keys for example policy.

I would rather go with 2nd option to add necessary IAM permissions. Hope it unblocks you.

On a side note, is there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<blabla>/version encrypted as it's not sensitive data and you will need to manage a key for that.

Thanks,
Ashish

[maddyexplore]

I tried both option but still failed,

❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

[ashishdhingra]

I tried both option but still failed,

❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Thanks,
Ashish

[ashishdhingra]

Investigation:

• Recreated parameter /cdk-bootstrap/hnb659fds/version as SecureString with value 20 (previous value for String parameter).
• Tried to redeploy stack. Got below error:

  ❌  TypescriptStack failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
      at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Deployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:5821)
      at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:199515)
      at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181237
  
   ❌ Deployment failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
      at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Deployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:5821)
      at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:199515)
      at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181237
  
  TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
  

• Modified bin\typescript.ts (project name is typescript):

  const app = new cdk.App();
  new TypescriptStack(app, 'TypescriptStack', { 
    synthesizer: new cdk.DefaultStackSynthesizer({
      generateBootstrapVersionRule: false
    })
  });

  This didn't generate the rule in CloudFormation template, but gave the below error during cdk deploy:

  ✨  Synthesis time: 4.49s
  
  
   ❌ Deployment failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
    at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Deployments.buildSingleAsset (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:10975)
    at async Object.buildAsset (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:197148)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181290
  
  TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
  

  • Reverted the changes made in previous step. Modified IAM role cdk-hnb659fds-deploy-role-<ACCOUNT_ID>-<REGION> to add the following permission:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "VisualEditor0",
              "Effect": "Allow",
              "Action": [
                  "kms:Decrypt",
                  "kms:Encrypt",
                  "kms:GenerateDataKey"
              ],
              "Resource": "<KMS_KEY_ARN>"
          }
      ]
  }

  Running cdk deploy still gave the above error.

Finding:
Following stack trace, it appears error is thrown at at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/api/deployments.ts:665:13). Code path is buildSingleAsset() > validateBootstrapStackVersion() > validateVersion() > versionFromSsmParameter().

The method versionFromSsmParameter() uses AWS JS SDK SSM.getParameter() without using the WithDecryption flag/option. Hence, encrypted value for SecureString is retrieved from SSM (in other words, it is not decrypted automatically).

The document Read Systems Manager values at synthesis time mentions that Only plain Systems Manager strings may be retrieved. Secure strings cannot be retrieved. The latest version will always be returned. Specific versions cannot be requested..

Need to discuss with Core CLI team.

[maddyexplore]

I tried both option but still failed,
❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Even though its not a sensitive data, we thought that storing it in a secure way since SSM parameter supports the SecureString

would appreciate the quicker fix for this

[ashishdhingra]

I tried both option but still failed,
❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Even though its not a sensitive data, we thought that storing it in a secure way since SSM parameter supports the SecureString

would appreciate the quicker fix for this

@maddyexplore Good morning. Per review from the CDK team, this is not an issue. Resources created by bootstrap template should not be manually altered. Doing so may lead to unexpected failures, as observed in this issue.

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.