CodePipeline limits

[azamin]

Hello,

I am seriously worried about the CodePipeline limits.
Let please somebody check my math and give me a hint how can I configure ADF to manage the baseline in 80 production accounts each of which operate in multiple (different) regions.

A pipeline can have maximum of 10 stages and 50 actions in a stage, this means theoretical maximum of 500 actions.
Two of the stages are always reserved for CodeCommit and CodeBuild, so it's actually 400 actions.
Each regional deployment needs 2 actions (replace->execute), so we go down to 200 account/regions per pipeline.

Because of the hard limit of 50 actions per stage (as per AWS Support), we can have maximum of 25 account/regions per stage.
This can be something between one account with 25 regions or 25 accounts with one region.
A stage in ADF represents also an AWS Organizaton Unit, which means also maximum of 25 accounts per OU.

The only workaround that I found is to use multiple pipelines to the same repository, but we still have the limit to 25 accounts per region, so we have to separate them in OU buckets on the Organization
like this:
regions: eu-west-1
/customers1/
/customers2/
..
regions: eu-central-1
/customers1/
/customers2/

Looks ugly and because of limit the 200 regional deployments per pipeline, we can have maximum of 8 OU buckets with maximum of 25 accounts per bucket.
Please let somebody tell me that I am wrong here, because I don't see how these 80 accounts will scale to 300 so that we can move our SaaS to AWS.

Thank you

[marck-oemar]

I've literally came across this exact problem 10 minutes ago, moving a bunch (more than 25) of accounts into an OU. Extremely disappointing.

I'm going to request limit increase and find out what their hard limits are. Documentation says it might take up to 2 weeks.

[azamin]

@elmoenco, for me it was only 3 working days and their support employee answered:
"I have an update from the service team and they informed that the maximum actions in a stage that we support is 50, which is the hard limit for our system. "

Good luck with your case and please let me know if you got it increased so that escalate to our account manager.

[marck-oemar]

Thanks! This is very bad. I can scale as far as I want with tools like concourse. This requires an abstracted approach of pipelines stages and jobs, all because of CodePipeline limitations.

…

On 13 Aug 2019, at 14:19, azamin ***@***.***> wrote: @elmoenco, for me it was only 3 working days and their support employee answered: "I have an update from the service team and they informed that the maximum actions in a stage that we support is 50, which is the hard limit for our system. " Good luck with your case and please let me know if you got it increased so that escalate to our account manager. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[azamin]

i love ADF and want to continue using it, so I decided for this workaround:
Please let me know if there is better way

# aws-deployment-framework-pipelines/deployment_maps/baseline-guardduty.yml
pipelines:
 - name: baseline-guardduty
   type: cc-cloudformation
   contains_transform: true # because it's SAM
   params:
     - SourceAccountId: xxxDEPLOYMENTxxx
     - NotificationEndpoint: xxx@xxxx
     - Image: "aws/codebuild/standard:2.0"
   targets:
     - path: xxxMASTERxxx # TODO: replace with either the Deplyment or the Security account
       regions: eu-central-1 # the main region 

#  baseline-guardduty/template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Configures Guardduty within the managed OUs in our Organization
Metadata:
  License: Apache-2.0
Resources:
  baselineGuardDuty:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: python3.7
      CodeUri: .
      Description: Configures Guardduty within the managed OUs in our Organization
      MemorySize: 256
      Timeout: 900
      Events:
        DailyExecution:
          Type: Schedule
          Properties:
            Schedule: rate(1 day)
      Policies:
        - Statement:
          - Sid: ListAccountsAndRegions
            Effect: Allow
            Action:
              - organizations:ListAccountsForParent
              - ec2:DescribeRegions
            Resource: '*'
        - Statement:
          - Sid: DeployGuardDuty 
            Effect: Allow
            Action:
              - sts:AssumeRole
            Resource: 'arn:aws:iam::*:role/OrganizationAccountAccessRole' # TODO: replace with a limited role 

# baseline-guardduty/lambda_function.py 
def lambda_handler(event, context):
    managed_accounts = get_managed_accounts()
    session_master = assume(GD_MASTER_ACCOUNT)
    ec2 = boto3.client('ec2', 'eu-central-1')
    regions = [region['RegionName'] for region in ec2.describe_regions()['Regions']]
    for region in regions:
        detector_id_master = get_or_create_detector(session_master, region, GD_MASTER_ACCOUNT)
        members = get_members(session_master, region, detector_id_master)
        for account_id, account in managed_accounts.items():
            #logger.info("working in %s for %s", region, account['Name'])
            if account_id in [GD_MASTER_ACCOUNT, ORGANIZATION_ACCOUNT]:
                continue # because we can't assume this role and the accounts are already provisioned
            session_sub = assume(account_id)
            detector_id_sub = get_or_create_detector(session_sub, region, account_id)
            if detector_id_sub not in members.keys():
                create_and_invite(session_master, region, detector_id_master, account_id, account['Email'])
                list_and_accept(session_sub, region, detector_id_sub)

It loops over 500 regions for about 4 minutes every day.
Same approach also for the AWS Config, but there loop first on accounts and then regions

[eliran-allcloud]

Hi,
you can create a pipeline for each region from the same source code.
so you can have 400 accounts in each region

pipelines:
  - name: sample-vpc-eu-west-1
    type: cc-cloudformation
    regions: eu-west-1
    deployment_role: infra-deployment-role
    params: &generic_params
      - SourceAccountId: 111111111111
      - RepositoryName: sample-vpc # Here we are inserting the RepositoryName as a Parameter
    targets:
      - /banking/testing
      - approval
      - /banking/production

  - name: sample-vpc-us-east-1
    type: cc-cloudformation
    regions: us-east-1
    deployment_role: infra-deployment-role
    params: *generic_params # Using The YAML Anchors concept as mentioned above, we get those same values.
    targets:
      - /banking/testing
      - approval
      - /banking/production

[azamin]

Thank you @eliran-allcloud for opening issue #103 - it was an awesome idea!

The maximum number of accounts/region deployments for the most common pipeline type is 200, because each of them needs two actions. See the template cc-cloudformation.yml.j2

{% for region in stage.regions %}
{% if action %}
- Name: {{ stage.name }}-{{ region }}-replace  # for ActionMode change_set_replace
- Name: {{ stage.name }}-{{ region }}-execute  # for ActionMode change_set_execute

I just opened PR #137 , because it's important that we be honest by letting early adopters know and understand CodePipeline limits.

[bundyfx]

needs two actions

You don't need to have two actions, however it is the default. Take a look into the docs about the action: property you can pass in, and, the different options it allows.

[azamin]

changing the action mode increases the default limit from 200 to 400 accounts per region, which seems OK for me. Thanks

[Wildpipe]

For anyone stumbling upon this thread there is now a wave option available that will automatically split the stages when they reach the limit. The option is documented here: https://github.com/awslabs/aws-deployment-framework/blob/master/docs/user-guide.md#targets-syntax

[sbkok]

Thank you for your patience. I am happy to inform you that this limit is resolved with the release of Wave deployments as part of release v3.2.0 just now.
Please open a new issue if you are experiencing any issues related to this feature.