(codepipeline) cross-region-stack-* causes cdk diff to fail for users who are only allowed to assume the lookup-role

[lestephane]

Describe the bug

I've been unable to cdk diff my cdk app since I've added another region (to deploy certificates to us-east-1), because I follow AWS best practices of least privileged access, and only have the permission to assume the lookup-role in all the account / region pairs I need to work with. And BootstraplessSynthesizer does not use the lookup-role at all.

Expected Behavior

• The lookup-role is assumed to determine the diff for the cross-support stack FIRST and the deploy-role is then assumed as fallback. Like it's done elsewhere.
• cdk diff succeeds

Current Behavior

• The deploy-role is assumed first
• The cdk diff fails

Reproduction Steps

• deploy a cdkpipeline (IacPipeline) in iac account A1 region R1 (R1 being geographically close, such as eu-central-1 for german customers), having first cdk bootstrap-ed A1R1 using privileged credentials.
• add locally (on a dev machine) a new stack definition for another account A2 region R2 (R2 being us-east-1as is often required for cloudfront ceritifcates), having first cdk bootstrap-ed A2R2 using privileged credentials. The stack is not deployed yet.
• the dev credentials are that of an A1 account role (iac-dev-role) whose only permissions are to allow AssumeRole of the cdk lookup-role for A1R1 and A2R2
• a (least privileged) dev user who can only assume the lookup-role A1R1 and A2R2 runs cdk diff '**'
• the command fails:

Stack cross-region-stack-A1:R2 (IacPipeline-support-R2)
current credentials could not be used to assume 'arn:aws:iam::A1:role/cdk-hnb659fds-deploy-role-A1-R1', but are for the right account. Proceeding anyway.

User: arn:aws:sts::A1:assumed-role/iac-dev-role/... is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:R2:A1:stack/IacPipeline-support-us-east-1/* because no identity-based policy allows the cloudformation:DescribeStacks action

Possible Solution

cdk bootstrap could be extended with a --needed flag to output all the AxRy account+region pairs where the bootstrap still needs to run. That would improve the ergonomics a bit. I realize it's impossible to figure out which role to assume in each account to check for bootstrapping. But there is at least OrganizationAccountAccessRole you could try to assume in each target account if the current credentials allow.

Having this would have made it clear that I do NOT need to cdk bootstrap A1R2, because of the BootstraplessSynthesizer

Once I realize I do NOT need to cdk bootstrap A1R2, it becomes clear that it is the bootstrap synthesizer that is to blame, because it did not attempt to assume cdk-hnb659fds-lookup-role-A1-R1 first.

It took me a few hours to figure out all this. And the use of a BootstraplessSynthesizer looks like a code smell. Why is it not OK to just include A1Ry (for each foreign region y that the pipeline intends to deploy to) as needing to be bootstrapped? That would allow you to keep a consistent behaviour everywhere. And you would then be able to focus on how to enable bootstrapping all these AxRy pairs more efficiently. The current behaviour is confusing.

Additional Information/Context

No response

CDK CLI Version

2.83.0 (build 0fd7f2b)

Framework Version

No response

Node.js Version

v16.19.0

OS

Ubuntu 22.04.2 LTS

Language

Typescript

Language Version

[email protected]

Other information

No response

[peterwoodworth]

Can you please expand on the steps you took, I'm a bit fuzzy on the details.

• The error message you pasted is showing A1:R2, but you never describe trying to create an A1:R2 stack, what stack exactly is failing?
• You mention BootstraplessSynthesizer a few times. Can you please show how you're using it, and clarify if you're using it when this behavior occurs?
• Can you share the full cdk bootstrap commands you're running?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[lestephane]

I can try to boil it down, because the 2-3 h I spent coming up with the initial description of the problem (based on the code review I did on the CDK) seems to confuse rather than help.

Two facts:

1. i have the permission to assume the lookup role in all account / region pairs involved in my app (including the account / region of the cross-region-stack- (which cdk creates automatically through some badly documented hack). So all regions and accounts are bootstrapped.

2. I want to perform a cdk diff which is a read-only operation, and for this, it should be sufficient to be able to assume the lookup-role everywhere. I do have the permission to assume lookup-roles in all account / region pairs that were bootstrapped.

Based on 1. and 2., I should be able to do the cdk diff, but I'm not. The CDK cli insists on assuming deploy-role for the cross-region-* support stack.

As @rix0rrr pointed out, the cross-region part is a mess, so It's not like I'm coming out of the blue with this.

I'm showing you what happens when technical debt has accumulated: I can't do a CDK diff when I should be able to, in a cross-region scenario.