(ECR): (Custom::ECRAutoDeleteImages fails on repo rename)

[duffzm]

Describe the bug

Within the same deployment environment I perform cdk deploy for ECR repositories and images stacks

1. using code from a default branch and then
2. I run it again (after successful deployment) to a feature branch.

The default branch stacks deploy successfully but the feature branch deployment fails to update my stacks due to what seems to be a renaming issue on ECR repositories in connection with Custom::ECRAutoDeleteImages resources . This is my first error

The following resource(s) failed to update: [basimagerepoAutoDeleteImagesCustomResource9BC77A26].

Because of

Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: User: arn:aws:sts::1234567890:assumed-role/my-stack-prefix-CustomECRAutoDeleteImage-1ESL2R8Y9CLAL/my-stack-prefix-infrastr-CustomECRAutoDeleteImage-D9sNwuezjpai is not authorized to perform: ecr:DescribeRepositories on resource: arn:aws:ecr:us-east-1:543276908693:repository/my/base_image because no identity-based policy allows the ecr:DescribeRepositories action at throwDefaultError (/var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22) at deserializeAws_json1_1DescribeRepositoriesCommandError (/var/runtime/node_modules/@aws-sdk/client-ecr/dist-cjs/protocols/Aws_json1_1.js:1212:51) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24 at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20 at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46) at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22 at async f (/var/task/index.js:1:3557) at async Runtime.handler (/var/task/index.js:1:1152) (RequestId: af95f9de-6958-4e02-af53-84a6a54d9b2c)

and then, when the rollback is attempted, the rollback fails on

The following resource(s) failed to update: [baseimagerepoAutoDeleteImagesCustomResource9BC77A26].

because of

Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: User: arn:aws:sts::1234567890:assumed-role/my-stack-prefix-CustomECRAutoDeleteImage-1ESL2R8Y9CLAL/my-stack-prefix-CustomECRAutoDeleteImage-D9sNwuezjpai is not authorized to perform: ecr:DescribeRepositories on resource: arn:aws:ecr:us-east-1:1234567890:repository/my/new/base_image because no identity-based policy allows the ecr:DescribeRepositories action at throwDefaultError (/var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22) at deserializeAws_json1_1DescribeRepositoriesCommandError (/var/runtime/node_modules/@aws-sdk/client-ecr/dist-cjs/protocols/Aws_json1_1.js:1212:51) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24 at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20 at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46) at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22 at async f (/var/task/index.js:1:3557) at async Runtime.handler (/var/task/index.js:1:1152) (RequestId: a6111bf7-1f99-4a5c-b9c4-224a039a04a0)

Note 1. that the the two error messages are different in that the initial failure includes my old repo name but the rollback failure includes the new name. Here they are side-by-side. The first one is UPDATE_FAILED, the second is UPDATE_ROLLBACK_FAILED

...User: arn:aws:sts::1234567890:assumed-role/my-stack-prefix-CustomECRAutoDeleteImage-1ESL2R8Y9CLAL/my-stack-prefix-CustomECRAutoDeleteImage-D9sNwuezjpai is not authorized to perform: ecr:DescribeRepositories on resource: arn:aws:ecr:us-east-1:1234567890:repository/my/base_image
...User: arn:aws:sts::1234567890:assumed-role/my-stack-prefix-CustomECRAutoDeleteImage-1ESL2R8Y9CLAL/my-stack-prefix-CustomECRAutoDeleteImage-D9sNwuezjpai is not authorized to perform: ecr:DescribeRepositories on resource: arn:aws:ecr:us-east-1:1234567890:repository/my/new/base_image

Note2. that all the newly named repos/images I created in ECR appeare to have deployed successfully, even though some them belonged to the failed stacks. So the issue seems to be isolated to the Custom::ECRAutoDeleteImages resource.

Note3. Termination protection is not enabled on these tasks or repositories and specifically we have on our ECR construct.

removal_policy = aws_cdk.RemovalPolicy.DESTROY
auto_delete_images = True

Expected Behavior

1. Previously named repos would be destroyed along with any images they house
2. New repos/images successful
3. Custom::ECRAutoDeleteImages updated successfully

Current Behavior

See description section. Full errors logged there.

Reproduction Steps

cdk deploy with

props = {}
props["repository_name"] = "name1"
props["removal_policy"] = aws_cdk.RemovalPolicy.DESTROY
props["auto_delete_images"] = True

Repository(
    scope=scope,
    id=construct_id,
    **props,
    **kwargs,
)

then presumably another deploy (in same stack) with

props = {}
props["repository_name"] = "name2"
props["removal_policy"] = aws_cdk.RemovalPolicy.DESTROY
props["auto_delete_images"] = True

Repository(
    scope=scope,
    id=construct_id,
    **props,
    **kwargs,
)

Possible Solution

Race condition with custom resource settings needs resolve.

Additional Information/Context

None

CDK CLI Version

2.2.200

Framework Version

2.89.0

Node.js Version

16

OS

Linux

Language

Python

Language Version

3.10.11

Other information

No response

[pahud]

I guess it's because the AutoDeleteImagesCustomResourc provider is only allowed to ecr:BatchDeleteImage on name1, when the repository is updated like that, the repo name will be changed to name2 and the custom resource can't describe images on name1 because name1 does not exist any more.

A quick hack is to allow the provider to describe all images but maybe we have better solutions.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.