Cloud Custodian

---

News Corp customizations

To avoid any unnecessary code conflict while merging upstream changes, we must follow the rule Do NOT change the code unless necessary.

• If things can be done without customizing Custodian, please do not customize it. e.g. You may leverage features like invoke-lambda, webhook, value_from of value filter etc. to integrate with your app
• Add new lines of code over revising existing code/logic
• Do NOT format existing code (keep it as it is please)
• Please spare some time to create a PR for upstream if you've completed a bug fix (to return to the community)

c7n

1. c7n

   • c7n - enable --vars option to load vars file
   • aws - reduce jitter of retry

2. ami resource

   • aws - ami - add last-launched-time filter

3. ebs resource

   • aws - divide into small batches when getting EBS resource

4. manager

   • aws - add c7n_resource_type_id to resource response

5. notify action

   • aws - prepare iam-saml-provider for notify and more important for security hub

6. policy

   • aws - introduce tag:custodian-policy as a version to avoid massive re-deployments
   • aws - support metrics keyword in policy to reduce the cost of custom metrics
   • policy - rename property comments to title

7. security hub

   • aws-finding - support Id param and auto-gen value: filter finding support Id param; generate id with policy name instead of policy data; generate the finding ID for those resources do not support tags

8. service quota resource

   • service quota - bugfix TooManyRequestsException when calling the ListServices in us-east-1
   • service quota - bugfix usage-metric requests more than 1440 data points

9. tag action

• aws - support marking missing resource when copy related tag
• aws - ec2 - add property 'delete' to rename-tag to achieve copy-tag effect

11. value filter

• filter value - support GCP labelisation when extracting content from value_from
• normalise value_type apply to keys in value_from.expr
• add key_type to enable key normalization
  • make key_type apply to value_from.expr as well
  • support key_type in value_filter
• add annotation op to value filter
• enable variables in expr of value_from
• aws - value filter value_from support default_value so that can load resources.json from other policies result

11. webhook action

• webhook action supports os env variables

c7n-mailer

• add ServiceNow notification
• mailer - jira delivery

c7n_org

• support org level vars in config file
• c7n-org supports the argument not-accounts
• c7n-org - fix NoCredentialsError when getting creds from instance profile too frequently

c7n_gcp

1. gcp.project

   • enable annotation op for IAM policy in gcp.project
     • extract values from gcp.project iam policy
     • convert values to gcp lable

2. filters.iam-policy

   • iam-policy - user-role support roles and verb_argument() refactor

3. gcp.bucket

   • gcp-bucket - add set-labels and mark(ed)-for-op

---

Cloud Custodian is a rules engine for managing public cloud accounts and resources. It allows users to define policies to enable a well managed cloud infrastructure, that's both secure and cost optimized. It consolidates many of the adhoc scripts organizations have into a lightweight and flexible tool, with unified metrics and reporting.

Custodian can be used to manage AWS, Azure, and GCP environments by ensuring real time compliance to security policies (like encryption and access requirements), tag policies, and cost management via garbage collection of unused resources and off-hours resource management.

Custodian policies are written in simple YAML configuration files that enable users to specify policies on a resource type (EC2, ASG, Redshift, CosmosDB, PubSub Topic) and are constructed from a vocabulary of filters and actions.

It integrates with the cloud native serverless capabilities of each provider to provide for real time enforcement of policies with builtin provisioning. Or it can be run as a simple cron job on a server to execute against large existing fleets.

Cloud Custodian is a CNCF Sandbox project, lead by a community of hundreds of contributors.

Features

• Comprehensive support for public cloud services and resources with a rich library of actions and filters to build policies with.
• Supports arbitrary filtering on resources with nested boolean conditions.
• Dry run any policy to see what it would do.
• Automatically provisions serverless functions and event sources ( AWS CloudWatchEvents, AWS Config Rules, Azure EventGrid, GCP AuditLog & Pub/Sub, etc)
• Cloud provider native metrics outputs on resources that matched a policy
• Structured outputs into cloud native object storage of which resources matched a policy.
• Intelligent cache usage to minimize api calls.
• Supports multi-account/subscription/project usage.
• Battle-tested - in production on some very large cloud environments.

Links

• Homepage
• Docs
• Developer Install
• Presentations

Quick Install

$ python3 -m venv custodian
$ source custodian/bin/activate
(custodian) $ pip install c7n

Usage

The first step to using Cloud Custodian is writing a YAML file containing the policies that you want to run. Each policy specifies the resource type that the policy will run on, a set of filters which control resources will be affected by this policy, actions which the policy with take on the matched resources, and a mode which controls which how the policy will execute.

The best getting started guides are the cloud provider specific tutorials.

• AWS Getting Started
• Azure Getting Started
• GCP Getting Started

As a quick walk through, below are some sample policies for AWS resources.

1. will enforce that no S3 buckets have cross-account access enabled.
2. will terminate any newly launched EC2 instance that do not have an encrypted EBS volume.
3. will tag any EC2 instance that does not have the follow tags "Environment", "AppId", and either "OwnerContact" or "DeptID" to be stopped in four days.

policies:
 - name: s3-cross-account
   description: |
     Checks S3 for buckets with cross-account access and
     removes the cross-account access.
   resource: aws.s3
   region: us-east-1
   filters:
     - type: cross-account
   actions:
     - type: remove-statements
       statement_ids: matched

 - name: ec2-require-non-public-and-encrypted-volumes
   resource: aws.ec2
   description: |
    Provision a lambda and cloud watch event target
    that looks at all new instances and terminates those with
    unencrypted volumes.
   mode:
    type: cloudtrail
    role: CloudCustodian-QuickStart
    events:
      - RunInstances
   filters:
    - type: ebs
      key: Encrypted
      value: false
   actions:
    - terminate

 - name: tag-compliance
   resource: aws.ec2
   description: |
     Schedule a resource that does not meet tag compliance policies to be stopped in four days. Note a separate policy using the`marked-for-op` filter is required to actually stop the instances after four days.
   filters:
    - State.Name: running
    - "tag:Environment": absent
    - "tag:AppId": absent
    - or:
      - "tag:OwnerContact": absent
      - "tag:DeptID": absent
   actions:
    - type: mark-for-op
      op: stop
      days: 4

You can validate, test, and run Cloud Custodian with the example policy with these commands:

# Validate the configuration (note this happens by default on run)
$ custodian validate policy.yml

# Dryrun on the policies (no actions executed) to see what resources
# match each policy.
$ custodian run --dryrun -s out policy.yml

# Run the policy
$ custodian run -s out policy.yml

You can run Cloud Custodian via Docker as well:

# Download the image
$ docker pull cloudcustodian/c7n
$ mkdir output

# Run the policy
#
# This will run the policy using only the environment variables for authentication
$ docker run -it \
  -v $(pwd)/output:/home/custodian/output \
  -v $(pwd)/policy.yml:/home/custodian/policy.yml \
  --env-file <(env | grep "^AWS\|^AZURE\|^GOOGLE") \
  cloudcustodian/c7n run -v -s /home/custodian/output /home/custodian/policy.yml

# Run the policy (using AWS's generated credentials from STS)
#
# NOTE: We mount the ``.aws/credentials`` and ``.aws/config`` directories to
# the docker container to support authentication to AWS using the same credentials
# credentials that are available to the local user if authenticating with STS.

$ docker run -it \
  -v $(pwd)/output:/home/custodian/output \
  -v $(pwd)/policy.yml:/home/custodian/policy.yml \
  -v $(cd ~ && pwd)/.aws/credentials:/home/custodian/.aws/credentials \
  -v $(cd ~ && pwd)/.aws/config:/home/custodian/.aws/config \
  --env-file <(env | grep "^AWS") \
  cloudcustodian/c7n run -v -s /home/custodian/output /home/custodian/policy.yml

The custodian cask tool is a go binary that provides a transparent front end to docker that mirors the regular custodian cli, but automatically takes care of mounting volumes.

Consult the documentation for additional information, or reach out on gitter.

Cloud Provider Specific Help

For specific instructions for AWS, Azure, and GCP, visit the relevant getting started page.

• AWS
• Azure
• GCP

Get Involved

• GitHub - (This page)
• Slack - Real time chat if you're looking for help or interested in contributing to Custodian!
  • Gitter - (Older real time chat, we're likely migrating away from this)
• Mailing List - Our project mailing list, subscribe here for important project announcements, feel free to ask questions
• Reddit - Our subreddit
• StackOverflow - Q&A site for developers, we keep an eye on the cloudcustodian tag
• YouTube Channel - We're working on adding tutorials and other useful information, as well as meeting videos

Community Resources

We have a regular community meeting that is open to all users and developers of every skill level. Joining the mailing list will automatically send you a meeting invite. See the notes below for more technical information on joining the meeting.

• Community Meeting Videos
• Community Meeting Notes Archive
• Upcoming Community Events
• Cloud Custodian Annual Report 2021 - Annual health check provided to the CNCF outlining the health of the project

Additional Tools

The Custodian project also develops and maintains a suite of additional tools here https://github.com/cloud-custodian/cloud-custodian/tree/master/tools:

• Org: Multi-account policy execution.

• PolicyStream: Git history as stream of logical policy changes.

• Salactus: Scale out s3 scanning.

• Mailer: A reference implementation of sending messages to users to notify them.

• Trail Creator: Retroactive tagging of resources creators from CloudTrail

• TrailDB: Cloudtrail indexing and time series generation for dashboarding.

• LogExporter: Cloud watch log exporting to s3

• Cask: Easy custodian exec via docker

• Guardian: Automated multi-account Guard Duty setup

• Omni SSM: EC2 Systems Manager Automation

• Mugc: A utility used to clean up Cloud Custodian Lambda policies that are deployed in an AWS environment.

Contributing

See https://cloudcustodian.io/docs/contribute.html

Security

If you've found a security related issue, a vulnerability, or a potential vulnerability in Cloud Custodian please let the Cloud Custodian Security Team know with the details of the vulnerability. We'll send a confirmation email to acknowledge your report, and we'll send an additional email when we've identified the issue positively or negatively.

Code of Conduct

This project adheres to the CNCF Code of Conduct

By participating, you are expected to honor this code.