(cdk deploy): cdk deploy fails when shared assets are deployed to different accounts because Docker assets are not built locally for all targets #26446
Labels
@aws-cdk/assets
Related to the @aws-cdk/assets package
bug
This issue is a bug.
effort/medium
Medium work item – several days of effort
p2
Comments
igirardi
added
bug
github-actions
bot
added
the
@aws-cdk/assets
|
I receive a similar error when using publishing a single docker image to multiple regions in the same account via a TarballImageAsset construct. Regions: us-east-1 and us-west-2 Error: Using an older version of the CDK ^2.70.0 fixes the issue as previously mentioned by @igirardi. However, I want to use the latest CDK version for other reasons. |
|
Related: #25962 |
|
Thank you @tmokmss for the insights #25962 (comment) |
pahud
added
p1
effort/medium
|
As a workaround I have cleared all docker cache locally via the following commands which forces to rebuild images and push to AWS ECR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
What is the issue?
In version
2.87.0of theAWS CDK CLIDocker assets are not properly built locally and assets push toAmazon ECRfails. This happens when Docker assets are shared between stacks that are deployed to different accounts.Deployment fails with the following error:
Deployment with version
2.79.1is fine. These findings are related to a change in the asset deployment with recentAWS CLIversions.What is the impact?
If a
CDKapp contains two stacks deployed to different accounts that use the same asset, thecdk deploycommand would fail because local assets are not built for all target accounts.Who is affected?
Recent
CDKversions for example version2.87.0.How do I resolve this?
Work around with the recent
AWS CDK CLIis to force asset build using theextraHashfunctionality. This work around can be complex for customers deploying several stacks sharing assets. Good choice for theextraHashparameter string is thetarget account idto avoid unnecessary rebuild of assets deployed to the same account.Customers need to explicitly force asset build with extra flags while this was the default behavior with older
AWS CDK CLIversions.Additional information
This ticket has been opened as documentation for customers having similar issues and to guide for a proper work around. It would be nice to restore the previous
AWS CDKfunctionality or provide an explanation of the reasons of this change.Expected Behavior
Running
cdk deployshould create/update CloudFormation stacks.Current Behavior
The following is the output from a
cdk deployrun that should have triggeredcdkto create the app's stacks in CloudFormation.Reproduction Steps
CDKapp folder structure:Content of
jest.config.js:Content of
tsconfig.json:{ "compilerOptions": { "target": "ES2020", "module": "commonjs", "lib": [ "es2020", "dom" ], "declaration": true, "strict": true, "noImplicitAny": true, "strictNullChecks": true, "noImplicitThis": true, "alwaysStrict": true, "noUnusedLocals": false, "noUnusedParameters": false, "noImplicitReturns": true, "noFallthroughCasesInSwitch": false, "inlineSourceMap": true, "inlineSources": true, "experimentalDecorators": true, "strictPropertyInitialization": false, "typeRoots": [ "./node_modules/@types" ] }, "exclude": [ "node_modules", "cdk.out" ] }Content of
package.json:{ "name": "container-deploy-issue-typescript", "version": "0.1.0", "bin": { "container-deploy-issue-typescript": "bin/container-deploy-issue-typescript.js" }, "scripts": { "build": "tsc", "watch": "tsc -w", "test": "jest", "cdk": "cdk" }, "devDependencies": { "@types/jest": "^29.5.1", "@types/node": "20.1.7", "jest": "^29.5.0", "ts-jest": "^29.1.0", "aws-cdk": "2.87.0", "ts-node": "^10.9.1", "typescript": "~5.1.3" }, "dependencies": { "aws-cdk-lib": "2.87.0", "constructs": "^10.0.0", "source-map-support": "^0.5.21" } }Content of
source/index.js:Content of
cdk.json:{ "app": "npx ts-node --prefer-ts-exts bin/cdk-app.ts", "watch": { "include": [ "**" ], "exclude": [ "README.md", "cdk*.json", "**/*.d.ts", "**/*.js", "tsconfig.json", "package*.json", "yarn.lock", "node_modules", "test" ] }, "context": { "@aws-cdk/aws-lambda:recognizeLayerVersion": true, "@aws-cdk/core:checkSecretUsage": true, "@aws-cdk/core:target-partitions": [ "aws", "aws-cn" ], "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true, "@aws-cdk/aws-iam:minimizePolicies": true, "@aws-cdk/core:validateSnapshotRemovalPolicy": true, "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true, "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true, "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true, "@aws-cdk/aws-apigateway:disableCloudWatchRole": true, "@aws-cdk/core:enablePartitionLiterals": true, "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true, "@aws-cdk/aws-iam:standardizedServicePrincipals": true, "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true, "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true, "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true, "@aws-cdk/aws-route53-patters:useCertificate": true, "@aws-cdk/customresources:installLatestAwsSdkDefault": false, "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true, "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true, "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true, "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true, "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true, "@aws-cdk/aws-redshift:columnId": true, "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true, "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true, "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true, "@aws-cdk/aws-kms:aliasNameRef": true, "@aws-cdk/core:includePrefixInUniqueNameGeneration": true } }Content of
Dockerfile:Content of
bin/cdk-app.ts:Similar project and code structure is in the Reproduction Steps of issue #25714.
Run
npm installto generate thepackage-lock.jsonand install the npm packages locally.For the deployment to multiple target accounts make sure to execute
cdk bootstrapsetting the flags--trustand--cloudformation-execution-policies.Run
npx cdk deploy --all --require-approval neverto reproduce the issue.Possible Solution
Introduce
extraHashto force asset rebuild when same asset has to be locally built and deployed to multiple target accounts.Content of
bin/cdk-app.ts:Additional Information/Context
Test deployment with
AWS CDKversion2.79.1changing the version in the filepackage.json:{ "name": "container-deploy-issue-typescript", "version": "0.1.0", "bin": { "container-deploy-issue-typescript": "bin/container-deploy-issue-typescript.js" }, "scripts": { "build": "tsc", "watch": "tsc -w", "test": "jest", "cdk": "cdk" }, "devDependencies": { "@types/jest": "^29.5.1", "@types/node": "20.1.7", "jest": "^29.5.0", "ts-jest": "^29.1.0", "aws-cdk": "2.79.1", "ts-node": "^10.9.1", "typescript": "~5.1.3" }, "dependencies": { "aws-cdk-lib": "2.79.1", "constructs": "^10.0.0", "source-map-support": "^0.5.21" } }Install the packages
npm installand runnpx cdk deploy --all --require-approval neverto check that deployment is successful with the olderAWS CDKversion.CDK CLI Version
2.87.0 (build 9fca790)
Framework Version
No response
Node.js Version
v18.16.1
OS
OS X & Ubuntu & Amazon Linux 2
Language
Typescript
Language Version
5.1.6
Other information
Thanks for the hard work on CDK! I'm a very happy user.
The text was updated successfully, but these errors were encountered: