feat(migrate): Add CDK Migrate --from-scan functionality

[HBobertz]

cdk migrate

⚠️CAUTION⚠️: CDK Migrate is currently experimental and may have breaking changes in the future.

CDK Migrate generates a CDK app from deployed AWS resources using --from-scan, deployed AWS CloudFormation stacks using --from-stack, and local AWS CloudFormation templates using --from-path.

To learn more about the CDK Migrate feature, see Migrate to AWS CDK. For more information on cdk migrate command options, see cdk migrate command reference.

The new CDK app will be initialized in the current working directory and will include a single stack that is named with the value you provide using --stack-name. The new stack, app, and directory will all use this name. To specify a different output directory, use --output-path. You can create the new CDK app in any CDK supported programming language using --language.

Migrate from an AWS CloudFormation stack

Migrate from a deployed AWS CloudFormation stack in a specific AWS account and AWS Region using --from-stack. Provide --stack-name to identify the name of your stack. Account and Region information are retrieved from default CDK CLI sources. Use --account and --region options to provide other values. The following is an example that migrates myCloudFormationStack to a new CDK app using TypeScript:

$ cdk migrate --language typescript --from-stack --stack-name 'myCloudFormationStack'

Migrate from a local AWS CloudFormation template

Migrate from a local YAML or JSON AWS CloudFormation template using --from-path. Provide a name for the stack that will be created in your new CDK app using --stack-name. Account and Region information are retrieved from default CDK CLI sources. Use --account and --region options to provide other values. The following is an example that creates a new CDK app using TypeScript that includes a myCloudFormationStack stack from a local template.json file:

$ cdk migrate --language typescript --from-path "./template.json" --stack-name "myCloudFormationStack"

Migrate from deployed AWS resources

Migrate from deployed AWS resources in a specific AWS account and Region that are not associated with an AWS CloudFormation stack using --from-scan. These would be resources that were provisioned outside of an IaC tool. CDK Migrate utilizes the IaC generator service to scan for resources and generate a template. Then, the CDK CLI references the template to create a new CDK app. To learn more about IaC generator, see Generating templates for existing resources.

Account and Region information are retrieved from default CDK CLI sources. Use --account and --region options to provide other values. The following is an example that creates a new CDK app using TypeScript that includes a new myCloudFormationStack stack from deployed resources:

$ cdk migrate --language typescript --from-scan --stack-name "myCloudFormationStack"

Since CDK Migrate relies on the IaC generator service, any limitations of IaC generator will apply to CDK Migrate. For general limitations, see Considerations.

IaC generator limitations with discovering resource and property values will also apply here. As a result, CDK Migrate will only migrate resources supported by IaC generator. Some of your resources may not be supported and some property values may not be accessible. For more information, see Iac generator and write-only properties and Supported resource types.

You can specify filters using --filter to specify which resources to migrate. This is a good option to use if you are over the IaC generator total resource limit.

After migration, you must resolve any write-only properties that were detected by IaC generator from your deployed resources. To learn more, see Resolve write-only properties.

Examples

Generate a TypeScript CDK app from a local AWS CloudFormation template.json file

$ # template.json is a valid cloudformation template in the local directory
$ cdk migrate --stack-name MyAwesomeApplication --language typescript --from-path MyTemplate.json

This command generates a new directory named MyAwesomeApplication within your current working directory, and
then initializes a new CDK application within that directory. The CDK app contains a MyAwesomeApplication stack with resources configured to match those in your local CloudFormation template.

This results in a CDK application with the following structure, where the lib directory contains a stack definition
with the same resource configuration as the provided template.json.

├── README.md
├── bin
│   └── my_awesome_application.ts
├── cdk.json
├── jest.config.js
├── lib
│   └── my_awesome_application-stack.ts
├── package.json
├── tsconfig.json

Generate a Python CDK app from a deployed stack

If you already have a CloudFormation stack deployed in your account and would like to manage it with CDK, you can migrate the deployed stack to a new CDK app. The value provided with --stack-name must match the name of the deployed stack.

$ # generate a Python application from MyDeployedStack in your account
$ cdk migrate --stack-name MyDeployedStack --language python --from-stack

This will generate a Python CDK app which will synthesize the same configuration of resources as the deployed stack.

Generate a TypeScript CDK app from deployed AWS resources that are not associated with a stack

If you have resources in your account that were provisioned outside AWS IaC tools and would like to manage them with the CDK, you can use the --from-scan option to generate the application.

In this example, we use the --filter option to specify which resources to migrate. You can filter resources to limit the number of resources migrated to only those specified by the --filter option, including any resources they depend on, or resources that depend on them (for example A filter which specifies a single Lambda Function, will find that specific table and any alarms that may monitor it). The --filter argument offers both AND as well as OR filtering.

OR filtering can be specified by passing multiple --filter options, and AND filtering can be specified by passing a single --filter option with multiple comma separated key/value pairs as seen below (see below for examples). It is recommended to use the --filter option to limit the number of resources returned as some resource types provide sample resources by default in all accounts which can add to the resource limits.

--from-scan takes 3 potential arguments: --new, most-recent, and undefined. If --new is passed, CDK Migrate will initiate a new scan of the account and use that new scan to discover resources. If --most-recent is passed, CDK Migrate will use the most recent scan of the account to discover resources. If neither --new nor --most-recent are passed, CDK Migrate will take the most recent scan of the account to discover resources, unless there is no recent scan, in which case it will initiate a new scan.

# Filtering options
identifier|id|resource-identifier=<resource-specific-resource-identifier-value>
type|resource-type-prefix=<resource-type-prefix>
tag-key=<tag-key>
tag-value=<tag-value>

Additional examples of migrating from deployed resources

$ # Generate a typescript application from all un-managed resources in your account
$ cdk migrate --stack-name MyAwesomeApplication --language typescript --from-scan

$ # Generate a typescript application from all un-managed resources in your account with the tag key "Environment" AND the tag value "Production"
$ cdk migrate --stack-name MyAwesomeApplication --language typescript --from-scan --filter tag-key=Environment,tag-value=Production

$ # Generate a python application from any dynamoDB resources with the tag-key "dev" AND the tag-value "true" OR any SQS::Queue
$ cdk migrate --stack-name MyAwesomeApplication --language python --from-scan --filter type=AWS::DynamoDb::,tag-key=dev,tag-value=true --filter type=SQS::Queue

$ # Generate a typescript application from a specific lambda function by providing it's specific resource identifier
$ cdk migrate --stack-name MyAwesomeApplication --language typescript --from-scan --filter identifier=myAwesomeLambdaFunction

CDK Migrate Limitations

• CDK Migrate does not currently support nested stacks, custom resources, or the Fn::ForEach intrinsic function.

• CDK Migrate will only generate L1 constructs and does not currently support any higher level abstractions.

• CDK Migrate successfully generating an application does not guarantee the application is immediately deployable.
  It simply generates a CDK application which will synthesize a template that has identical resource configurations
  to the provided template.

  • CDK Migrate does not interact with the CloudFormation service to verify the template
    provided can deploy on its own. This means CDK Migrate will not verify that any resources in the provided
    template are already managed in other CloudFormation templates, nor will it verify that the resources in the provided
    template are available in the desired regions, which may impact ADC or Opt-In regions.

  • If the provided template has parameters without default values, those will need to be provided
    before deploying the generated application.

In practice this is how CDK Migrate generated applications will operate in the following scenarios:

Situation	Result
Provided template + stack-name is from a deployed stack in the account/region	The CDK application will deploy as a changeset to the existing stack
Provided template has no overlap with resources already in the account/region	The CDK application will deploy a new stack successfully
Provided template has overlap with Cloudformation managed resources already in the account/region	The CDK application will not be deployable unless those resources are removed
Provided template has overlap with un-managed resources already in the account/region	The CDK application will not be deployable until those resources are adopted with cdk import
No template has been provided and resources exist in the region the scan is done	The CDK application will be immediatly deployable and will import those resources into a new cloudformation stack upon deploy

The provided template is already deployed to CloudFormation in the account/region

If the provided template came directly from a deployed CloudFormation stack, and that stack has not experienced any drift,
then the generated application will be immediately deployable, and will not cause any changes to the deployed resources.
Drift might occur if a resource in your template was modified outside of CloudFormation, namely via the AWS Console or AWS CLI.

The provided template is not deployed to CloudFormation in the account/region, and there is not overlap with existing resources in the account/region

If the provided template represents a set of resources that have no overlap with resources already deployed in the account/region,
then the generated application will be immediately deployable. This could be because the stack has never been deployed, or
the application was generated from a stack deployed in another account/region.

In practice this means for any resource in the provided template, for example,

    "S3Bucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketName": "MyBucket",
        "AccessControl": "PublicRead",
      },
      "DeletionPolicy": "Retain"
    }

There must not exist a resource of that type with the same identifier in the desired region. In this example that identfier
would be "MyBucket"

The provided template is not deployed to CloudFormation in the account/region, and there is overlap with existing resources in the account/region

If the provided template represents a set of resources that overlap with resources already deployed in the account/region,
then the generated application will not be immediately deployable. If those overlapped resources are already managed by
another CloudFormation stack in that account/region, then those resources will need to be manually removed from the provided
template. Otherwise, if the overlapped resources are not managed by another CloudFormation stack, then first remove those
resources from your CDK Application Stack, deploy the cdk application successfully, then re-add them and run cdk import
to import them into your deployed stack.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

[HBobertz]

[INTEG TEST::cdk migrate typescript deploys successfully] Starting (pid 26740)...
[INTEG TEST::cdk migrate typescript deploys successfully] Done (123818 ms).
[INTEG TEST::cdk migrate python deploys successfully] Starting (pid 26740)...
[INTEG TEST::cdk migrate python deploys successfully] Done (111815 ms).
[INTEG TEST::cdk migrate csharp deploys successfully] Starting (pid 26740)...
[INTEG TEST::cdk migrate csharp deploys successfully] Done (110802 ms).
[INTEG TEST::cdk migrate java deploys successfully] Starting (pid 26740)...
[INTEG TEST::cdk migrate java deploys successfully] Done (110135 ms).
[INTEG TEST::cdk migrate generates migrate.json] Starting (pid 26740)...
[INTEG TEST::cdk migrate generates migrate.json] Done (27986 ms).
[INTEG TEST::cdk migrate --from-scan with AND/OR filters correctly filters resources] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-scan with AND/OR filters correctly filters resources] Done (425320 ms).
[INTEG TEST::cdk migrate --from-scan for resources with Write Only Properties generates warnings] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-scan for resources with Write Only Properties generates warnings] Done (435061 ms).
[INTEG TEST::cdk migrate --from-stack creates deployable typescript app] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-stack creates deployable typescript app] Done (124690 ms).
[INTEG TEST::cdk migrate --from-stack creates deployable python app] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-stack creates deployable python app] Done (135417 ms).
[INTEG TEST::cdk migrate --from-stack creates deployable csharp app] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-stack creates deployable csharp app] Done (133364 ms).
[INTEG TEST::cdk migrate --from-stack creates deployable java app] Starting (pid 26740)...
[INTEG TEST::cdk migrate --from-stack creates deployable java app] Done (132794 ms).
[INTEG TEST::test migrate deployment for app with localfile source in migrate.json] Starting (pid 26740)...
[INTEG TEST::test migrate deployment for app with localfile source in migrate.json] Done (135253 ms).
PASS tests/cli-integ-tests/cli.integtest.js (2008.601 s)
● Console

console.log
  Using regions: us-east-1

  at log (../../lib/with-aws.ts:36:11)

Test Suites: 2 skipped, 1 passed, 1 of 3 total
Tests: 80 skipped, 12 passed, 92 total
Snapshots: 0 total
Time: 2009.576 s

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: b2af1e4
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[tmokmss]

Congrats 🎉

Provided template has overlap with un-managed resources already in the account/region | The CDK application will not be deployable until those resources are adopted with cdk import

Isn't this a good use case of the new simplified import? #28060