-
Notifications
You must be signed in to change notification settings - Fork 149
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1606 from rundeck/cost-management
Cost Management Solution
- Loading branch information
Showing
7 changed files
with
348 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
104 changes: 104 additions & 0 deletions
104
docs/learning/solutions/cost-management/aws-list-unused-lambda.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # AWS - Identify Unused Lambda Functions | ||
|
|
||
| ## Description | ||
|
|
||
| This automation job generates a listing of AWS Lambda functions and highlights any that may be eligible for deletion based on modification and execution dates provided as job inputs. It provides a detailed report of Lambda functions, including their last modified and last execution dates, and recommends whether to keep or delete each function. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner. | ||
| - This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions. | ||
| - AWS CLI installed on the runner node | ||
| - jq tool for JSON parsing installed on the runner node | ||
| - Proper AWS credentials configured on the runner node | ||
|
|
||
| ## AWS IAM Permissions | ||
|
|
||
| The AWS IAM role or user associated with this job requires the following permissions: | ||
|
|
||
| - `lambda:ListFunctions` | ||
| - `logs:DescribeLogGroups` | ||
| - `logs:DescribeLogStreams` | ||
|
|
||
| These permissions should be applied to all resources (`"Resource": "*"`). | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "lambda:ListFunctions", | ||
| "logs:DescribeLogGroups", | ||
| "logs:DescribeLogStreams" | ||
| ], | ||
| "Resource": "*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## Job Options | ||
|
|
||
| | Option Name | Description | Default Value | | ||
| |------------------|-----------------------------------------------------------|---------------| | ||
| | `Region` | AWS region to query for Lambda functions | N/A | | ||
| | `Execution Date` | List functions that have not been called since this date | N/A | | ||
| | `Modified Date` | List functions older than this date | N/A | | ||
|
|
||
|
|
||
| ## Job Workflow | ||
|
|
||
| 1. The job runs on a node with the tag "RUNNER" | ||
| 2. It uses the AWS CLI to list all Lambda functions in the specified region | ||
| 3. For each function, it retrieves: | ||
| - The last modified date | ||
| - The last execution date (from CloudWatch Logs) | ||
| 4. It compares these dates against the provided execution and modification thresholds | ||
| 5. The job generates a report for each function, including: | ||
| - Function name | ||
| - Last modified date | ||
| - Last execution date | ||
| - Recommendation to keep or delete the function | ||
|
|
||
| ## Output | ||
|
|
||
| The job produces a detailed report with the following information for each Lambda function: | ||
|
|
||
| - Function name | ||
| - Last modified date | ||
| - Last execution date | ||
| - Recommendation: "Delete" or "Keep" | ||
|
|
||
| The recommendation output is color-coded for easy reading: | ||
| - Red background: Functions recommended for deletion | ||
| - Green background: Functions recommended to keep | ||
|
|
||
| ## Script Details | ||
|
|
||
| The job uses a Bash script to perform the following tasks: | ||
|
|
||
| 1. Set up variables for the AWS region and date thresholds | ||
| 2. Convert input dates to Unix timestamps and ISO 8601 format | ||
| 3. List all Lambda functions in the specified region | ||
| 4. For each function: | ||
| - Retrieve the last modified date | ||
| - Check for associated CloudWatch Logs | ||
| - Retrieve the last execution date from logs (if available) | ||
| - Compare dates against thresholds | ||
| - Generate a recommendation | ||
|
|
||
| ## Notes | ||
|
|
||
| - The job does not actually delete any functions; it only provides recommendations | ||
| - Functions are recommended for deletion if both the last modified date and the last execution date are earlier than the provided thresholds | ||
| - If a function has no associated CloudWatch Logs, its last execution date will be shown as "No logs found" | ||
| - If a function has logs but no executions, its last execution date will be shown as "No execution found" | ||
| - The script is designed to work on both Linux and macOS systems | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| If you encounter issues running this job: | ||
| 1. Ensure that the AWS CLI and jq are properly installed on the runner node | ||
| 2. Verify that the AWS credentials on the runner node have the necessary permissions |
98 changes: 98 additions & 0 deletions
98
docs/learning/solutions/cost-management/aws-list-unused-securitygroups.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # AWS - Identify Unused Security Groups | ||
|
|
||
| ## Description | ||
|
|
||
| This automation job generates a listing of AWS security groups that are not associated with any network interfaces and are therefore eligible for deletion. It checks various AWS services to ensure comprehensive coverage. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner. | ||
| - This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions. | ||
| - AWS CLI installed on the runner node. | ||
| - Proper AWS credentials configured on the runner node. | ||
|
|
||
| ## AWS IAM Permissions | ||
|
|
||
| The AWS IAM role or user associated with this job requires the following permissions: | ||
|
|
||
| - `ec2:DescribeSecurityGroups` | ||
| - `ec2:DescribeNetworkInterfaces` | ||
| - `elb:DescribeLoadBalancers` | ||
| - `elbv2:DescribeLoadBalancers` | ||
| - `rds:DescribeDBInstances` | ||
| - `elasticache:DescribeCacheClusters` | ||
| - `redshift:DescribeClusters` | ||
|
|
||
| These permissions should be applied to all resources in the specified region. | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "ec2:DescribeSecurityGroups", | ||
| "ec2:DescribeNetworkInterfaces", | ||
| "elb:DescribeLoadBalancers", | ||
| "elbv2:DescribeLoadBalancers", | ||
| "rds:DescribeDBInstances", | ||
| "elasticache:DescribeCacheClusters", | ||
| "redshift:DescribeClusters" | ||
| ], | ||
| "Resource": "*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## Job Options | ||
|
|
||
| | Option Name | Description | Default Value | | ||
| |-------------|-------------|---------------| | ||
| | `region` | AWS region to query for security groups | N/A | | ||
| | `always-show-results` | Show results even when checking AWS services results in Access Errors | false | | ||
|
|
||
| ## Job Workflow | ||
|
|
||
| 1. It uses the AWS CLI to list all security groups in the specified region. | ||
| 2. The script then checks for security groups associated with: | ||
| - Network interfaces | ||
| - Classic load balancers | ||
| - Application/Network load balancers | ||
| - RDS instances | ||
| - ElastiCache clusters | ||
| - Redshift clusters | ||
| 3. It compares the list of all security groups against those associated with the above services. | ||
| 4. The job generates a report of security groups that are not associated with any of these services and are eligible for deletion. | ||
|
|
||
| ## Output | ||
|
|
||
| The job produces a detailed report with the following information: | ||
|
|
||
| - List of all security groups in the region | ||
| - List of security groups associated with various AWS services | ||
| - Security groups that can be safely deleted (not associated with any service) | ||
| - Warnings for default security groups (which cannot be deleted) | ||
|
|
||
| ## Script Details | ||
|
|
||
| The job uses a Bash script to perform the following tasks: | ||
|
|
||
| 1. Fetch all security groups in the specified region | ||
| 2. Retrieve security groups associated with various AWS services | ||
| 3. Compare the lists to identify unused security groups | ||
| 4. Generate a report of security groups eligible for deletion | ||
|
|
||
| ## Notes | ||
|
|
||
| - The job does not actually delete any security groups; it only provides recommendations. | ||
| - Default security groups are excluded from the deletion recommendations. | ||
| - The script includes error handling and can optionally show the recommendation results even if some AWS API calls result in errors. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| If you encounter issues running this job: | ||
| 1. Ensure that the AWS CLI is properly installed on the runner node | ||
| 2. Verify that the AWS credentials on the runner node have the necessary permissions | ||
| 3. Check the `always-show-results` option if you want to see partial results in case of API errors |
101 changes: 101 additions & 0 deletions
101
docs/learning/solutions/cost-management/aws-list-unused-vpcs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # AWS - Identify Unused VPCs | ||
|
|
||
| ## Description | ||
|
|
||
| This automation job generates a listing of AWS Virtual Private Clouds (VPCs) that are not associated with any resources and are therefore eligible for deletion. It checks various AWS services to ensure comprehensive coverage. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner. | ||
| - This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions. | ||
| - AWS CLI installed on the runner node. | ||
| - Proper AWS credentials configured on the runner node. | ||
|
|
||
| ## AWS IAM Permissions | ||
|
|
||
| The AWS IAM role or user associated with this job requires the following permissions: | ||
|
|
||
| - `ec2:DescribeVpcs` | ||
| - `ec2:DescribeInstances` | ||
| - `rds:DescribeDBInstances` | ||
| - `elb:DescribeLoadBalancers` | ||
| - `elbv2:DescribeLoadBalancers` | ||
| - `ec2:DescribeNatGateways` | ||
| - `ec2:DescribeVpnConnections` | ||
| - `ec2:DescribeTransitGatewayVpcAttachments` | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "ec2:DescribeVpcs", | ||
| "ec2:DescribeInstances", | ||
| "rds:DescribeDBInstances", | ||
| "elb:DescribeLoadBalancers", | ||
| "elbv2:DescribeLoadBalancers", | ||
| "ec2:DescribeNatGateways", | ||
| "ec2:DescribeVpnConnections", | ||
| "ec2:DescribeTransitGatewayVpcAttachments" | ||
| ], | ||
| "Resource": "*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| These permissions should be applied to all resources in the specified region. | ||
|
|
||
| ## Job Options | ||
|
|
||
| | Option Name | Description | Default Value | | ||
| |----|----|----| | ||
| | `region` | AWS region to query for VPCs | N/A | | ||
| | `always-show-results` | Show results even when checking AWS services results in Access Errors | false | | ||
|
|
||
| ## Job Workflow | ||
|
|
||
| 1. It uses the AWS CLI to list all VPCs in the specified region. | ||
| 2. The script then checks for VPCs associated with: | ||
| - EC2 instances | ||
| - RDS instances | ||
| - Classic load balancers | ||
| - Application/Network load balancers | ||
| - NAT Gateways | ||
| - VPN Connections | ||
| - Transit Gateway attachments | ||
| 3. It compares the list of all VPCs against those associated with the above services. | ||
| 4. The job generates a report of VPCs that are not associated with any of these services and are eligible for deletion. | ||
|
|
||
| ## Output | ||
|
|
||
| The job produces a detailed report with the following information: | ||
|
|
||
| - List of all VPCs in the region | ||
| - List of VPCs associated with various AWS services | ||
| - VPCs that can be safely deleted (not associated with any service) | ||
| - Warnings for default VPCs (which cannot be deleted) | ||
|
|
||
| ## Script Details | ||
|
|
||
| The job uses a Bash script to perform the following tasks: | ||
|
|
||
| 1. Fetch all VPCs in the specified region | ||
| 2. Retrieve VPCs associated with various AWS services | ||
| 3. Compare the lists to identify unused VPCs | ||
| 4. Generate a report of VPCs eligible for deletion | ||
|
|
||
| ## Notes | ||
|
|
||
| - The job does not actually delete any VPCs; it only provides recommendations. | ||
| - Default VPCs are excluded from the deletion recommendations. | ||
| - The script includes error handling and can optionally show the recommendation results even if some AWS API calls result in errors. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| If you encounter issues running this job: | ||
| 1. Ensure that the AWS CLI is properly installed on the runner node | ||
| 2. Verify that the AWS credentials on the runner node have the necessary permissions | ||
| 3. Check the `always-show-results` option if you want to see partial results in case of API errors |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Cost Management | ||
|
|
||
| ### Overview | ||
|
|
||
| Leveraging Runbook Automation for Cost Management tasks in cloud environments can significantly improve efficiency and reduce operational costs. Such tools allow teams to create, schedule, and manage complex workflows across multiple cloud platforms, ensuring consistent execution of cost-saving measures. By automating these tasks, organizations can minimize human error, save time, and maintain better control over their cloud resources. Regular, automated cleanup processes help prevent unnecessary charges from idle or forgotten resources, optimize infrastructure usage, and enforce cost governance policies. Additionally, automation tools often provide role-based access control and audit trails, enhancing security and compliance. This makes them invaluable for organizations looking to streamline their cloud cost management efforts, regardless of the specific cloud provider or infrastructure setup. | ||
|
|
||
| ### Use Cases | ||
|
|
||
| - **Identify and Remove Unused Resources**: Identifying and removing unused resources in cloud solutions saves money by eliminating unnecessary costs associated with idle or forgotten assets that continue to incur charges without providing any value to the organization. | ||
| - **Right-size Computing Resources**: Automated resizing of compute resources in cloud solutions optimizes costs by dynamically adjusting capacity to match actual workload demands, ensuring you're not overpaying for underutilized resources or suffering performance issues due to undersized instances. | ||
|
|
||
|
|
||
| ### Prebuilt Automation | ||
| PagerDuty provides a solution that helps users start automating diagnostics quickly. This Solution consists of **prebuilt Automation Jobs** that show how the use cases above can be implemented in your environment. (Note: Some of these solutions may exist in other Solution Packages.) | ||
|
|
||
|
|
||
| | Examples | | ||
| | --- | | ||
| | <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused VPCs](/learning/solutions/cost-management/aws-list-unused-vpcs.md) | | ||
| | <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused Lambda Functions](/learning/solutions/cost-management/aws-list-unused-lambda.md) | | ||
| | <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused Security Groups](/learning/solutions/cost-management/aws-list-unused-securitygroups.md) | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters