Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pipelines] UpdatePipeline stage of new empty pipeline fails due to missing permissions (ref. #9227) #9606

Closed
MamishIo opened this issue Aug 11, 2020 · 8 comments
Assignees
Labels
@aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.

Comments

@MamishIo
Copy link

MamishIo commented Aug 11, 2020

I'm attempting to set up an initial blank pipeline via Java11, and have the same permissions issue noted in issue #9227, where the UpdatePipeline stage fails to due missing CloudFormation permissions needed for cdk deploy.

Reproduction Steps

Deploy empty (no application stages) pipeline from https://github.com/HtyCorp/cdk-repro-1

Requires: Java 11 (tested: Corretto-11.0.8.10.1), Maven (tested: 3.6.0), must have a Secrets Manager secret "GitHubRepoAccessToken" in account containing GitHub OAuth token for CodePipeline.

git clone https://github.com/HtyCorp/cdk-repro-1.git && cd cdk-repro-1 && mvn clean install && cdk synth && cdk deploy --require-approval=never

What did you expect to happen?

Default pipeline stages (Source -> Build -> UpdatePipeline) of deployed pipeline all execute successfully since no deployments or changes are required.

Role generated by CdkPipeline construct has permission to invoke CloudFormation, or is used by UpdatePipeline to assume account's 'cdk--deploy-role-' role for CloudFormation permissions.

What actually happened?

Pipeline is successfully created but UpdatePipeline stage is unable to execute cdk deploy due to missing cloudformation:GetTemplate permission:

User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee

Detailed CodeBuild logs:

[Container] 2020/08/11 14:25:11  BUILD: 1 commands
[Container] 2020/08/11 14:25:11 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2020/08/11 14:25:11 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:11 Entering phase INSTALL
[Container] 2020/08/11 14:25:11 Running command npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk
+ [email protected]
added 216 packages from 186 contributors in 7.207s

[Container] 2020/08/11 14:25:25 Phase complete: INSTALL State: SUCCEEDED
[Container] 2020/08/11 14:25:25 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:25 Entering phase PRE_BUILD
[Container] 2020/08/11 14:25:25 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2020/08/11 14:25:25 Phase context status code:  Message: 
[Container] 2020/08/11 14:25:25 Entering phase BUILD
[Container] 2020/08/11 14:25:25 Running command cdk -a . deploy CdkPipeline --require-approval=never --verbose
CDK toolkit version: 1.57.0 (build 2ccfc50)
Command line arguments: { _: [ 'deploy' ],
  a: '.',
  app: '.',
  'require-approval': 'never',
  requireApproval: 'never',
  verbose: 1,
  v: 1,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  '$0': '/usr/local/bin/cdk',
  STACKS: [ 'CdkPipeline' ],
  stacks: [ 'CdkPipeline' ] }
merged settings: { versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: '.',
  context: {},
  tags: [],
  assetMetadata: true,
  requireApproval: 'never',
  toolkitBucket: {},
  staging: true }
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
Resolving default credentials
Looking up default account ID from STS
Default account ID: 111122223333
Setting "CDK_DEFAULT_ACCOUNT" environment variable to 111122223333
context: { 'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true }
--app points to a cloud assembly, so we bypass synth
CdkPipeline: deploying...
Waiting for stack CDKToolkit to finish creating or updating...
CdkPipeline: checking if we can skip deploy
Call failed: getTemplate({"StackName":"CdkPipeline","TemplateStage":"Original"}) => User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee

 ❌  CdkPipeline failed: AccessDenied: User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
AccessDenied: User: arn:aws:sts::111122223333:assumed-role/CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE/AWSCodeBuild-d8a8414e-ce0f-40bf-bb9c-8b4526bf6ed6 is not authorized to perform: cloudformation:GetTemplate on resource: arn:aws:cloudformation:ap-southeast-2:111122223333:stack/CdkPipeline/81fa2090-dbdd-11ea-9fba-0266790261ee
    at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

[Container] 2020/08/11 14:25:27 Command did not exit successfully cdk -a . deploy CdkPipeline --require-approval=never --verbose exit status 1
[Container] 2020/08/11 14:25:27 Phase complete: BUILD State: FAILED
[Container] 2020/08/11 14:25:27 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk -a . deploy CdkPipeline --require-approval=never --verbose. Reason: exit status 1
[Container] 2020/08/11 14:25:27 Entering phase POST_BUILD
[Container] 2020/08/11 14:25:27 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2020/08/11 14:25:27 Phase context status code:  Message: 

Policy of generated IAM role CdkPipeline-DeploymentPipelineUpdatePipelineSelfMu-DC3BOF9ZJLNE (not modified in CDK app):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-southeast-2:111122223333:log-group:/aws/codebuild/CDKDeploymentPipeline-selfupdate",
                "arn:aws:logs:ap-southeast-2:111122223333:log-group:/aws/codebuild/CDKDeploymentPipeline-selfupdate:*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "codebuild:CreateReportGroup",
                "codebuild:CreateReport",
                "codebuild:UpdateReport",
                "codebuild:BatchPutTestCases"
            ],
            "Resource": "arn:aws:codebuild:ap-southeast-2:111122223333:report-group/CDKDeploymentPipeline-selfupdate-*",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:*:iam::*:role/*-deploy-role-*",
                "arn:*:iam::*:role/*-publishing-role-*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "cloudformation:DescribeStacks",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "s3:ListBucket",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::cdkpipeline-deploymentpipelineartifactsbucketf7fe-abcd12341a7n9",
                "arn:aws:s3:::cdkpipeline-deploymentpipelineartifactsbucketf7fe-abcd12341a7n9/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:ap-southeast-2:111122223333:key/abcd1234-e992-436c-b98d-4dbdea13b3df",
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": "arn:aws:kms:ap-southeast-2:111122223333:key/abcd1234-e992-436c-b98d-4dbdea13b3df",
            "Effect": "Allow"
        }
    ]
}

Environment

  • CLI Version : 1.57.0 (build 2ccfc50)
  • Framework Version: 1.57.0
  • Node.js Version: v10.22.0
  • OS : Ubuntu 18.04.4 LTS
  • Language (Version): Java (11)

Other


This is 🐛 Bug Report

@MamishIo MamishIo added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 11, 2020
@github-actions github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label Aug 11, 2020
@MamishIo MamishIo changed the title [pipelines] Initial pipeline UpdatePipeline stage fails due to missing permissions (ref. #9227) [pipelines] UpdatePipeline stage of new empty pipeline fails due to missing permissions (ref. #9227) Aug 11, 2020
@MamishIo
Copy link
Author

MamishIo commented Aug 12, 2020

Edit: Turns out this was my fault: cdk.json was misformatted, so new-style stack synthesis wasn't enabled. After correcting that, the UpdatePipeline cdk deploy command is correctly using the account's bootstrapped deployment role for the previously failing actions.

For posterity, fixed by changing:

{
  "app": "mvn -e -q exec:java",
  "output": "target/cloud-assembly",
  "@aws-cdk/core:newStyleStackSynthesis": "true"
}

to:

{
  "app": "mvn -e -q exec:java",
  "output": "target/cloud-assembly",
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": true
  }
}

@jonny-rimek
Copy link

@MamishIo thanks for posting your solution, just ran into the exact same problem <3

@OperationalFallacy
Copy link

Nice finding, @MamishIo!
Solved the same issue in 5 min while preparing to spend an hour :)

@briansunter
Copy link

You saved me with this fix

@matwerber1
Copy link
Contributor

matwerber1 commented May 15, 2021

Even after changing cdk.json and re-bootstrapping, I'm getting same error... my pipeline's mutate role is still missing the required CloudFormation:GetTemplate permissions. Investigating, will share if I find the problem.

@cbrgm
Copy link

cbrgm commented Jun 3, 2021

@matwerber1 any updates? I just ran into the same problem.

@matwerber1
Copy link
Contributor

matwerber1 commented Jun 3, 2021

@cbrgm yes, actually found my problem.

The issue was that I was correctly updating cdk.json with the "@aws-cdk/core:newStyleStackSynthesis": true flag, but the mistake was that I was locally running cdk deploy afterward to test the changes.

Being new to CDK pipelines, I didn't realize this was the mistake. With the way the pipeline runs, its actually going to pull the latest changes from your source code repository when it gets to the relevant build scripts - and since I had not committed and pushed the change to cdk.json to remote, it was still building based on the old cdk.json that was missing the new stack synthesis flag.

Lesson learned for me is that, with CDK pipelines, after your very first local cdk deploy to get the pipeline up and running, you should rely on pushing commits to the repo for future changes.

Hopefully this helps you?

@cbrgm
Copy link

cbrgm commented Jun 3, 2021

@matwerber1 Unbelievable! You just made my day! Thank you very much, that's what it was ...

scubbo added a commit to scubbo/rolledATwenty that referenced this issue Nov 5, 2021
Based on [this issue](aws/aws-cdk#9606), a older
versions of CDK don't correctly grant permissions for updates, which is an error
I'm seeing now.

I don't see `"@aws-cdk/core:newStyleStackSynthesis": true` in `cdk.json` here,
but...  I'm trying! I'll manually add it if this fails too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/pipelines CDK Pipelines library bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

7 participants