‼️ (cli): import fails with with 'S3 error: Access Denied #31716

EHadoux · 2024-10-10T14:20:50Z

Please add your +1 👍 to let us know you have encountered this

Status: IN-PROGRESS

Overview:

Versions 2.161.0, 2.161.1 and 2.162.0 of the CDK CLI will fail with an S3 error when performing an interactive cdk import.

In #31597 we changed cdk diff to always use the file asset publishing role, instead of direct CLI credentials. This included a refactor that impacted cdk import, which was now not uploading the stack template at all anymore. The operation that is now broken only happens in a case with interactive input, which is why this wasn't caught by integ tests.

Complete Error Message:

S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Workaround:

We are rolling out a fix now. If you are blocked by this, you can temporarily try using an older version of the CDK CLI to perform the import npx [email protected] import ...

Related Issues:

None yet.

Original report

Describe the bug

When using cdk import, the command tries to read a template that doesn't exist from the assets bucket.
It also returns an uninformative S3 error.

Regression Issue

Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I can import without any manual intervention (especially error-prone tinkering with the assets bucket).

Current Behavior

Output of import --force -v command:

[....]
[14:51:24] Reading existing template for stack Prod/Companies.
[14:51:24] Retrieved account ID XXXXXXXXX from disk cache
[14:51:24] Assuming role 'arn:aws:iam::XXXXXXXXX:role/cdk-hnb659fds-lookup-role-XXXXXXXXX-eu-west-2'.
Ignoring updated/deleted resources (--force): LAMBDAS APPARENTLY WITH UPDATES EVEN THOUGH THEY ARE NOT IN THE DIFF
[14:51:25] Retrieving template summary for stack Prod/Companies.
[14:51:25] Retrieved account ID XXXXXXXXX from disk cache
[14:51:25] Assuming role 'arn:aws:iam::XXXXXXXXX:role/cdk-hnb659fds-deploy-role-XXXXXXXXX-eu-west-2'.
[14:51:25] Call failed: getTemplateSummary({"TemplateURL":"https://s3.eu-west-2.amazonaws.com/cdk-XXXX-assets-XXXXXXXXX-eu-west-2/52852392599fc7aa03a114e1052f31dc18d35712e91b343b2038748dfa86a257.json"}) => S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html (code=ValidationError)
[14:51:25] Reading cached notices from /Users/.../.cdk/cache/notices.json
S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
[14:51:25] ValidationError: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
    at Request.extractError (/Users/.../node_modules/aws-cdk/lib/index.js:439:46717)
    at Request.callListeners (/Users/.../node_modules/aws-cdk/lib/index.js:439:91820)
    at Request.emit (/Users/.../node_modules/aws-cdk/lib/index.js:439:91268)
    at Request.emit (/Users/.../node_modules/aws-cdk/lib/index.js:439:200231)
    at Request.transition (/Users/.../node_modules/aws-cdk/lib/index.js:439:193784)
    at AcceptorStateMachine.runTo (/Users/.../node_modules/aws-cdk/lib/index.js:439:158656)
    at /Users/.../node_modules/aws-cdk/lib/index.js:439:158986
    at Request.<anonymous> (/Users/.../node_modules/aws-cdk/lib/index.js:439:194076)
    at Request.<anonymous> (/Users/.../node_modules/aws-cdk/lib/index.js:439:200306)
    at Request.callListeners (/Users/.../node_modules/aws-cdk/lib/index.js:439:91988)

Diff output

Stack Prod/Companies
Resources
[+] AWS::DynamoDB::Table Companies/Lists/CompaniesListsTable/CompaniesListsTable ListsCompaniesListsTableAFA73CC8


✨  Number of stacks with differences: 1

If I take the result of synth and put it in the bucket with exactly the expected key (52852392599fc7aa03a114e1052f31dc18d35712e91b343b2038748dfa86a257.json in this case) it works.

Reproduction Steps

Just trying to import a table should do it.

export class MyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);
    this.table = new Table(this, 'Table', {
      ...props,
      removalPolicy: RemovalPolicy.RETAIN,
    });
  }
}

Possible Solution

Either don't read the template from S3 as it won't contain the latest version. We can't have a template for importing that's identical to the one deployed, otherwise the resources to import will have been created already.

Or here

aws-cdk/packages/aws-cdk/lib/api/deployments.ts

Lines 429 to 435 in c1b240e

    
           // Upload the template, if necessary, before passing it to CFN 
        
           const cfnParam = await makeBodyParameter( 
        
             stackArtifact, 
        
             resolvedEnvironment, 
        
             new AssetManifestBuilder(), 
        
             envResources, 
        
             stackSdk);

should upload the template first.

This function is called here

aws-cdk/packages/aws-cdk/lib/import.ts

Line 250 in c1b240e

private async resourceIdentifiers(): Promise<ResourceIdentifiers> {

in turn indirectly called by the import command.

Additional Information/Context

I know it's not an Access Denied error per se as I have full admin access and can diff/deploy/etc. It should be a 404 instead of a 403.

Also, my lambdas prevent importing because of metadata changes even though they don't appear in the diff. Probably related to #31677

CDK CLI Version

2.161.1 (build 0a606c9)

Framework Version

No response

Node.js Version

20.17.0

OS

Mac OS

Language

TypeScript

Language Version

5.6.2

Other information

No response

The text was updated successfully, but these errors were encountered:

ashishdhingra · 2024-10-10T21:57:31Z

Reproducible using customer provided code:

Created a CDK stack, first empty with no resources and then an S3 bucket:

import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as s3 from 'aws-cdk-lib/aws-s3';

export class CdkimporttestStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'MyExampleBucket');
  }
}

In AWS console, created a new DynamoDB table named TestTable with partition key as Id of type String.

Now modify the created CDK stack to include DynamoDB table:

import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as s3 from 'aws-cdk-lib/aws-s3';

export class CdkimporttestStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'MyExampleBucket');

    new dynamodb.Table(this, 'TestTable', {
      partitionKey: {
        name: 'Id',
        type: dynamodb.AttributeType.STRING,
      },
      removalPolicy: cdk.RemovalPolicy.RETAIN,
    });
  }
}

Run cdk import --verbose. It gives the below error:

[14:40:30] CDK toolkit version: 2.161.1 (build 0a606c9)
[14:40:30] Command line arguments: {
...
[14:40:30] Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
[14:40:30] Determining if we're on an EC2 instance.
[14:40:30] Does not look like an EC2 instance.
[14:40:30] Toolkit stack: CDKToolkit
[14:40:30] Setting "CDK_DEFAULT_REGION" environment variable to us-east-2
[14:40:30] Resolving default credentials
[14:40:32] Looking up default account ID from STS
[14:40:32] Default account ID: 139480602983
[14:40:32] Setting "CDK_DEFAULT_ACCOUNT" environment variable to ACCOUNT_ID
...
[14:40:32] env: {
  CDK_DEFAULT_REGION: 'us-east-2',
  CDK_DEFAULT_ACCOUNT: 'ACCOUNT_ID',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '38.0.1',
  CDK_CLI_VERSION: '2.161.1'
}
CdkimporttestStack
[14:40:36] Reading existing template for stack CdkimporttestStack.
[14:40:36] Retrieved account ID ACCOUNT_ID from disk cache
[14:40:36] Assuming role 'arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-lookup-role-ACCOUNT_ID-us-east-2'.
[14:40:37] Retrieving template summary for stack CdkimporttestStack.
[14:40:37] Retrieved account ID ACCOUNT_ID from disk cache
[14:40:37] Assuming role 'arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-deploy-role-ACCOUNT_ID-us-east-2'.
[14:40:38] Call failed: getTemplateSummary({"TemplateURL":"https://s3.us-east-2.amazonaws.com/cdk-hnb659fds-assets-ACCOUNT_ID-us-east-2/cb039627e5fca93282deb1f0f888242baafca890fe30d1efc0414471d978f6bf.json"}) => 
S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html (code=ValidationError)
[14:40:38] Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json
S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
[14:40:38] ValidationError: S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
    at Request.extractError (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:46717)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:91820)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:91268)
    at Request.emit (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:200231)
    at Request.transition (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:193784)
    at AcceptorStateMachine.runTo (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:158656)
    at /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:158986
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:194076)
    at Request.<anonymous> (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:200306)
    at Request.callListeners (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:439:91988)

Manually uploaded JSON file cb039627e5fca93282deb1f0f888242baafca890fe30d1efc0414471d978f6bf.json (which cdk import is trying to access, to the bootstrapped S3 bucket. The contents of the JSON file is the locally synthesized JSON template containing newly added DynamoDB table resource.

Ran cdk import --verbose again. It's successful.

[14:42:18] CDK toolkit version: 2.161.1 (build 0a606c9)
[14:42:18] Command line arguments: {
...
CdkimporttestStack
[14:42:24] Reading existing template for stack CdkimporttestStack.
[14:42:24] Retrieved account ID ACCOUNT_ID from disk cache
[14:42:24] Assuming role 'arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-lookup-role-ACCOUNT_ID-us-east-2'.
[14:42:26] Retrieving template summary for stack CdkimporttestStack.
[14:42:26] Retrieved account ID ACCOUNT_ID from disk cache
[14:42:26] Assuming role 'arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-deploy-role-ACCOUNT_ID-us-east-2'.
CdkimporttestStack/TestTable/Resource (AWS::DynamoDB::Table): enter TableName (empty to skip): TestTable
CdkimporttestStack: importing resources into stack...
[14:42:37] Retrieved account ID ACCOUNT_ID from disk cache
[14:42:37] Assuming role 'arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-deploy-role-ACCOUNT_ID-us-east-2'.
[14:42:37] CdkimporttestStack: checking if we can skip deploy
[14:42:38] CdkimporttestStack: template has changed
[14:42:38] CdkimporttestStack: deploying...
[14:42:38] Removing existing change set with name cdk-deploy-change-set if it exists
[14:42:38] Attempting to create ChangeSet with name cdk-deploy-change-set to update stack CdkimporttestStack
CdkimporttestStack: creating CloudFormation changeset...
[14:42:39] Initiated creation of changeset: arn:aws:cloudformation:us-east-2:139480602983:changeSet/cdk-deploy-change-set/fb75d55f-832b-462c-8e93-3207cf62b7d8; waiting for it to finish creating...
[14:42:39] Waiting for changeset cdk-deploy-change-set on stack CdkimporttestStack to finish creating...
[14:42:39] Changeset cdk-deploy-change-set on stack CdkimporttestStack is still creating
[14:42:45] Initiating execution of changeset arn:aws:cloudformation:us-east-2:139480602983:changeSet/cdk-deploy-change-set/fb75d55f-832b-462c-8e93-3207cf62b7d8 on stack CdkimporttestStack
[14:42:45] Execution of changeset arn:aws:cloudformation:us-east-2:139480602983:changeSet/cdk-deploy-change-set/fb75d55f-832b-462c-8e93-3207cf62b7d8 on stack CdkimporttestStack has started; waiting for the update to complete...
[14:42:45] Waiting for stack CdkimporttestStack to finish creating or updating...
[14:42:46] Stack CdkimporttestStack has an ongoing operation in progress and is not stable (IMPORT_IN_PROGRESS (User Initiated))
CdkimporttestStack | 0/3 | 2:42:45 PM | IMPORT_IN_PROGRESS   | AWS::CloudFormation::Stack | CdkimporttestStack User Initiated
CdkimporttestStack | 0/3 | 2:42:47 PM | IMPORT_IN_PROGRESS   | AWS::DynamoDB::Table | TestTable (TestTable5769773A) Resource import started.
CdkimporttestStack | 0/3 | 2:42:48 PM | IMPORT_IN_PROGRESS   | AWS::DynamoDB::Table | TestTable (TestTable5769773A) 
CdkimporttestStack | 1/3 | 2:42:48 PM | IMPORT_COMPLETE      | AWS::DynamoDB::Table | TestTable (TestTable5769773A) Resource import completed.
CdkimporttestStack | 1/3 | 2:42:49 PM | UPDATE_IN_PROGRESS   | AWS::DynamoDB::Table | TestTable (TestTable5769773A) Apply stack-level tags to imported resource if applicable.
CdkimporttestStack | 0/3 | 2:42:50 PM | UPDATE_COMPLETE      | AWS::DynamoDB::Table | TestTable (TestTable5769773A) 
CdkimporttestStack | 1/3 | 2:42:50 PM | IMPORT_COMPLETE      | AWS::CloudFormation::Stack | CdkimporttestStack 
[14:42:52] Stack CdkimporttestStack has completed updating

 ✅  CdkimporttestStack
Import operation complete. We recommend you run a drift detection operation to confirm your CDK app resource definitions are up-to-date. Read more here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/detect-drift-stack.html
[14:42:52] Reading cached notices from /Users/ashdhin/.cdk/cache/notices.json

References:

Didn't dive into root cause as of now.

awsdiegorad · 2024-10-10T23:12:15Z

I tested this issue using CDK CLI version 2.147.1 and was not able to reproduce it. However after updating to version 2.162.0 I was able to reproduce it, looks like there was a change to the import logic between these versions.

awsdiegorad · 2024-10-10T23:26:08Z

The last working version in my testing is 2.160.0. This issue started on CDK CLI version 2.161.0.

awsdiegorad · 2024-10-10T23:47:37Z

This change implemented in 2.161.0 stands out to me as a potential cause for this issue. This change moved where the synthesized template is uploaded from deployment.ts to cloudformation.ts .

rix0rrr · 2024-10-11T09:12:04Z

Thanks for the report! I'm looking into it.

github-actions · 2024-10-11T13:45:54Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

github-actions · 2024-10-11T13:45:55Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

In #31597 we changed `cdk diff` to always use the file asset publishing role, instead of direct CLI credentials. This included a refactor that impacted `cdk import`, which was now not uploading the stack template at all anymore. The operation that is now broken only happens in a case with interactive input, which is why this wasn't caught by integ tests. In this change, put the requisite asset-handling code around `makeBodyParameter` to make the asset uploading happen properly. In future PRs: - Add an integration test for `cdk import` which would have exposed the same error. - Refactor the contract of `makeBodyParameter`, and perhaps more around asset uploading, to make the expectations and promises of that function more clear; right now it was not obvious what the function would and wouldn't do for you, which led to this error. I did some refactorings in this PR already (renames, removing an unused argument). I saw an opportunity for more but didn't want to add risk and delay to this patch. Hence, forthcoming 😄 . Closes #31716. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

EHadoux added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 10, 2024

github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Oct 10, 2024

ashishdhingra self-assigned this Oct 10, 2024

ashishdhingra added needs-reproduction This issue needs reproduction. p2 and removed needs-triage This issue or PR still needs to be triaged. labels Oct 10, 2024

ashishdhingra added p1 effort/medium Medium work item – several days of effort and removed p2 needs-reproduction This issue needs reproduction. labels Oct 10, 2024

ashishdhingra removed their assignment Oct 10, 2024

rix0rrr mentioned this issue Oct 11, 2024

fix(cli): cdk import errors with 'S3 error: Access Denied' #31727

Merged

rix0rrr changed the title ~~(cli): import fails trying to read missing template in assets bucket~~ (cli): import fails with with 'S3 error: Access Denied Oct 11, 2024

rix0rrr changed the title ~~(cli): import fails with with 'S3 error: Access Denied~~ ‼️ (cli): import fails with with 'S3 error: Access Denied Oct 11, 2024

rix0rrr added management/tracking Issues that track a subject or multiple issues p0 labels Oct 11, 2024

rix0rrr pinned this issue Oct 11, 2024

mergify bot closed this as completed in #31727 Oct 11, 2024

mergify bot closed this as completed in cd324d0 Oct 11, 2024

github-actions bot locked as resolved and limited conversation to collaborators Oct 11, 2024

ashishdhingra removed the p1 label Oct 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

‼️ (cli): import fails with with 'S3 error: Access Denied #31716

‼️ (cli): import fails with with 'S3 error: Access Denied #31716

EHadoux commented Oct 10, 2024 •

edited by rix0rrr

Loading

ashishdhingra commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

rix0rrr commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

‼️ (cli): import fails with with 'S3 error: Access Denied #31716

‼️ (cli): import fails with with 'S3 error: Access Denied #31716

Comments

EHadoux commented Oct 10, 2024 • edited by rix0rrr Loading

Status: IN-PROGRESS

Overview:

Complete Error Message:

Workaround:

Related Issues:

Original report

Describe the bug

Regression Issue

Last Known Working CDK Version

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

ashishdhingra commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

awsdiegorad commented Oct 10, 2024

rix0rrr commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

EHadoux commented Oct 10, 2024 •

edited by rix0rrr

Loading