Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cost Management Solution #1606

Merged
merged 2 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/.vuepress/sidebar-menus/learning.ts
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,17 @@ export default [{
{ link: '/learning/solutions/storage-management/storage-list-largest-files.md', text: 'List Largest Files' },
{ link: '/learning/solutions/storage-management/storage-log-rotation.md', text: 'Log Rotation & Cleanup' },
]
},
{
text: 'Cost Management',
link: '/learning/solutions/cost-management/index.md',
collapsible: true,
children: [
{ link: '/learning/solutions/cost-management/index.md', text: 'Solution Summary' },
{ link: '/learning/solutions/cost-management/aws-list-unused-vpcs.md', text: 'AWS - Identify Unused VPCs' },
{ link: '/learning/solutions/cost-management/aws-list-unused-lambda.md', text: 'AWS - Identify Unused Lambda Functions' },
{ link: '/learning/solutions/cost-management/aws-list-unused-securitygroups.md', text: 'AWS - Identify Unused Security Groups' },
]
}
]
},
Expand Down
6 changes: 3 additions & 3 deletions docs/learning/solutions/containers/index.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Container Solutions

### Overview
Automating container management with Runbook Automation can significantly enhance your business operations by streamlining and simplifying the deployment, scaling, and maintenance of containerized applications. By leveraging Rundeck's powerful automation capabilities, you can reduce manual intervention, minimize human error, and ensure consistent and reliable execution of tasks. This leads to increased efficiency and productivity, as routine and repetitive tasks are handled automatically, freeing up your IT team to focus on more strategic initiatives.
Automating container management with Runbook Automation can significantly enhance your business operations by streamlining and simplifying the deployment, scaling, and maintenance of containerized applications. By leveraging powerful automation capabilities, you can reduce manual intervention, minimize human error, and ensure consistent and reliable execution of tasks. This leads to increased efficiency and productivity, as routine and repetitive tasks are handled automatically, freeing up your IT team to focus on more strategic initiatives.

### Use Cases
There are multiple use-cases and benefits to the Containers solution. Here are a few of the most common examples:

- **Scale Kubernetes Deployoments**: Integrate seamlessly with Kubernetes clusters, enabling DevOps teams to automate complex scaling tasks
- **Diganose issues by checking Pod Status and Errors**: Create jobs to automatically check pod status, retrieve logs, and identify errors at set intervals or in response to specific triggers
- **Scale Kubernetes Deployments**: Integrate seamlessly with Kubernetes clusters, enabling DevOps teams to automate complex scaling tasks
- **Diagnose issues by checking Pod Status and Errors**: Create jobs to automatically check pod status, retrieve logs, and identify errors at set intervals or in response to specific triggers
- **Gather recent logs from Pods**: Capture state and log information before restarting containers to help troubleshoot after the solution is back online.
- **Start, Kill, and Execute Commands in Containers**: Execute commands in containers to control environment related issues and isolate code execution.

Expand Down
104 changes: 104 additions & 0 deletions docs/learning/solutions/cost-management/aws-list-unused-lambda.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# AWS - Identify Unused Lambda Functions

## Description

This automation job generates a listing of AWS Lambda functions and highlights any that may be eligible for deletion based on modification and execution dates provided as job inputs. It provides a detailed report of Lambda functions, including their last modified and last execution dates, and recommends whether to keep or delete each function.

## Prerequisites

- Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner.
- This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions.
- AWS CLI installed on the runner node
- jq tool for JSON parsing installed on the runner node
- Proper AWS credentials configured on the runner node
fdevans marked this conversation as resolved.
Show resolved Hide resolved

## AWS IAM Permissions

The AWS IAM role or user associated with this job requires the following permissions:

- `lambda:ListFunctions`
- `logs:DescribeLogGroups`
- `logs:DescribeLogStreams`

These permissions should be applied to all resources (`"Resource": "*"`).

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:ListFunctions",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": "*"
}
]
}
```

## Job Options

| Option Name | Description | Default Value |
|------------------|-----------------------------------------------------------|---------------|
| `Region` | AWS region to query for Lambda functions | N/A |
| `Execution Date` | List functions that have not been called since this date | N/A |
| `Modified Date` | List functions older than this date | N/A |


## Job Workflow

1. The job runs on a node with the tag "RUNNER"
2. It uses the AWS CLI to list all Lambda functions in the specified region
3. For each function, it retrieves:
- The last modified date
- The last execution date (from CloudWatch Logs)
4. It compares these dates against the provided execution and modification thresholds
5. The job generates a report for each function, including:
- Function name
- Last modified date
- Last execution date
- Recommendation to keep or delete the function

## Output

The job produces a detailed report with the following information for each Lambda function:

- Function name
- Last modified date
- Last execution date
- Recommendation: "Delete" or "Keep"

The recommendation output is color-coded for easy reading:
- Red background: Functions recommended for deletion
- Green background: Functions recommended to keep

## Script Details

The job uses a Bash script to perform the following tasks:

1. Set up variables for the AWS region and date thresholds
2. Convert input dates to Unix timestamps and ISO 8601 format
3. List all Lambda functions in the specified region
4. For each function:
- Retrieve the last modified date
- Check for associated CloudWatch Logs
- Retrieve the last execution date from logs (if available)
- Compare dates against thresholds
- Generate a recommendation

## Notes

- The job does not actually delete any functions; it only provides recommendations
- Functions are recommended for deletion if both the last modified date and the last execution date are earlier than the provided thresholds
- If a function has no associated CloudWatch Logs, its last execution date will be shown as "No logs found"
- If a function has logs but no executions, its last execution date will be shown as "No execution found"
- The script is designed to work on both Linux and macOS systems

## Troubleshooting

If you encounter issues running this job:
1. Ensure that the AWS CLI and jq are properly installed on the runner node
2. Verify that the AWS credentials on the runner node have the necessary permissions
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# AWS - Identify Unused Security Groups

## Description

This automation job generates a listing of AWS security groups that are not associated with any network interfaces and are therefore eligible for deletion. It checks various AWS services to ensure comprehensive coverage.

## Prerequisites

- Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner.
- This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions.
- AWS CLI installed on the runner node.
- Proper AWS credentials configured on the runner node.

## AWS IAM Permissions

The AWS IAM role or user associated with this job requires the following permissions:

- `ec2:DescribeSecurityGroups`
- `ec2:DescribeNetworkInterfaces`
- `elb:DescribeLoadBalancers`
- `elbv2:DescribeLoadBalancers`
- `rds:DescribeDBInstances`
- `elasticache:DescribeCacheClusters`
- `redshift:DescribeClusters`

These permissions should be applied to all resources in the specified region.

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeSecurityGroups",
"ec2:DescribeNetworkInterfaces",
"elb:DescribeLoadBalancers",
"elbv2:DescribeLoadBalancers",
"rds:DescribeDBInstances",
"elasticache:DescribeCacheClusters",
"redshift:DescribeClusters"
],
"Resource": "*"
}
]
}
```

## Job Options

| Option Name | Description | Default Value |
|-------------|-------------|---------------|
| `region` | AWS region to query for security groups | N/A |
| `always-show-results` | Show results even when checking AWS services results in Access Errors | false |

## Job Workflow

1. It uses the AWS CLI to list all security groups in the specified region.
2. The script then checks for security groups associated with:
- Network interfaces
- Classic load balancers
- Application/Network load balancers
- RDS instances
- ElastiCache clusters
- Redshift clusters
3. It compares the list of all security groups against those associated with the above services.
4. The job generates a report of security groups that are not associated with any of these services and are eligible for deletion.

## Output

The job produces a detailed report with the following information:

- List of all security groups in the region
- List of security groups associated with various AWS services
- Security groups that can be safely deleted (not associated with any service)
- Warnings for default security groups (which cannot be deleted)

## Script Details

The job uses a Bash script to perform the following tasks:

1. Fetch all security groups in the specified region
2. Retrieve security groups associated with various AWS services
3. Compare the lists to identify unused security groups
4. Generate a report of security groups eligible for deletion

## Notes

- The job does not actually delete any security groups; it only provides recommendations.
- Default security groups are excluded from the deletion recommendations.
- The script includes error handling and can optionally show the recommendation results even if some AWS API calls result in errors.

## Troubleshooting

If you encounter issues running this job:
1. Ensure that the AWS CLI is properly installed on the runner node
2. Verify that the AWS credentials on the runner node have the necessary permissions
3. Check the `always-show-results` option if you want to see partial results in case of API errors
101 changes: 101 additions & 0 deletions docs/learning/solutions/cost-management/aws-list-unused-vpcs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# AWS - Identify Unused VPCs

## Description

This automation job generates a listing of AWS Virtual Private Clouds (VPCs) that are not associated with any resources and are therefore eligible for deletion. It checks various AWS services to ensure comprehensive coverage.

## Prerequisites

- Turn on "[Runner as Node](/administration/runner/runner-management/node-dispatch.html#runner-as-a-node)" setting on your Runner.
- This requires version 5.8.0 or higher. Adjustments to Node tab may be required for earlier versions.
- AWS CLI installed on the runner node.
- Proper AWS credentials configured on the runner node.

## AWS IAM Permissions

The AWS IAM role or user associated with this job requires the following permissions:

- `ec2:DescribeVpcs`
- `ec2:DescribeInstances`
- `rds:DescribeDBInstances`
- `elb:DescribeLoadBalancers`
- `elbv2:DescribeLoadBalancers`
- `ec2:DescribeNatGateways`
- `ec2:DescribeVpnConnections`
- `ec2:DescribeTransitGatewayVpcAttachments`

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVpcs",
"ec2:DescribeInstances",
"rds:DescribeDBInstances",
"elb:DescribeLoadBalancers",
"elbv2:DescribeLoadBalancers",
"ec2:DescribeNatGateways",
"ec2:DescribeVpnConnections",
"ec2:DescribeTransitGatewayVpcAttachments"
],
"Resource": "*"
}
]
}
```

These permissions should be applied to all resources in the specified region.

## Job Options

| Option Name | Description | Default Value |
|----|----|----|
| `region` | AWS region to query for VPCs | N/A |
| `always-show-results` | Show results even when checking AWS services results in Access Errors | false |

## Job Workflow

1. It uses the AWS CLI to list all VPCs in the specified region.
2. The script then checks for VPCs associated with:
- EC2 instances
- RDS instances
- Classic load balancers
- Application/Network load balancers
- NAT Gateways
- VPN Connections
- Transit Gateway attachments
3. It compares the list of all VPCs against those associated with the above services.
4. The job generates a report of VPCs that are not associated with any of these services and are eligible for deletion.

## Output

The job produces a detailed report with the following information:

- List of all VPCs in the region
- List of VPCs associated with various AWS services
- VPCs that can be safely deleted (not associated with any service)
- Warnings for default VPCs (which cannot be deleted)

## Script Details

The job uses a Bash script to perform the following tasks:

1. Fetch all VPCs in the specified region
2. Retrieve VPCs associated with various AWS services
3. Compare the lists to identify unused VPCs
4. Generate a report of VPCs eligible for deletion

## Notes

- The job does not actually delete any VPCs; it only provides recommendations.
- Default VPCs are excluded from the deletion recommendations.
- The script includes error handling and can optionally show the recommendation results even if some AWS API calls result in errors.

## Troubleshooting

If you encounter issues running this job:
1. Ensure that the AWS CLI is properly installed on the runner node
2. Verify that the AWS credentials on the runner node have the necessary permissions
3. Check the `always-show-results` option if you want to see partial results in case of API errors
21 changes: 21 additions & 0 deletions docs/learning/solutions/cost-management/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Cost Management

### Overview

Leveraging Runbook Automation for Cost Management tasks in cloud environments can significantly improve efficiency and reduce operational costs. Such tools allow teams to create, schedule, and manage complex workflows across multiple cloud platforms, ensuring consistent execution of cost-saving measures. By automating these tasks, organizations can minimize human error, save time, and maintain better control over their cloud resources. Regular, automated cleanup processes help prevent unnecessary charges from idle or forgotten resources, optimize infrastructure usage, and enforce cost governance policies. Additionally, automation tools often provide role-based access control and audit trails, enhancing security and compliance. This makes them invaluable for organizations looking to streamline their cloud cost management efforts, regardless of the specific cloud provider or infrastructure setup.

### Use Cases

- **Identify and Remove Unused Resources**: Identifying and removing unused resources in cloud solutions saves money by eliminating unnecessary costs associated with idle or forgotten assets that continue to incur charges without providing any value to the organization.
- **Right-size Computing Resources**: Automated resizing of compute resources in cloud solutions optimizes costs by dynamically adjusting capacity to match actual workload demands, ensuring you're not overpaying for underutilized resources or suffering performance issues due to undersized instances.


### Prebuilt Automation
PagerDuty provides a solution that helps users start automating diagnostics quickly. This Solution consists of **prebuilt Automation Jobs** that show how the use cases above can be implemented in your environment. (Note: Some of these solutions may exist in other Solution Packages.)


| Examples |
| --- |
| <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused VPCs](/learning/solutions/cost-management/aws-list-unused-vpcs.md) |
| <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused Lambda Functions](/learning/solutions/cost-management/aws-list-unused-lambda.md) |
| <img src="/assets/img/aws-logo.png" width="30" height="30"> [AWS - Identify Unused Security Groups](/learning/solutions/cost-management/aws-list-unused-securitygroups.md) |
11 changes: 10 additions & 1 deletion docs/learning/solutions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,16 @@ highlights:
- title: Storage Management
icon: hard-drive
details: Technology users and leaders face challenges with time-consuming and error-prone manual processes when managing storage across various systems. Without automation managing storage across complex systems requires significant effort and risks operational inefficiencies, and service disruptions.
link: /learning/solutions/storage-management/index.md
link: /learning/solutions/storage-management/index.md

- title: Cost Management
icon: money-check-dollar
details: Runbook Automation for Cost Management in cloud environments streamlines complex workflows, reduces operational costs, minimizes errors, and enhances efficiency by automating resource optimization, cleanup processes, and policy enforcement across multiple platforms.
link: /learning/solutions/cost-management/index.md

- title: Getting Started
icon: circle-play
details: Click here to learn how to get started with Runbook Automation prebuilt solutions.
link: /learning/solutions/getting-started.md

---
Loading