Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MWAA module and example #585

Merged
merged 8 commits into from
Jul 8, 2022
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions examples/mwaa/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
## Using Amazon MWAA with Amazon EKS

The example demonstrates how to use Amazon Managed Workflows for Apache Airflow (MWAA) with Amazon EKS.

This example was originated from the steps provided on MWAA documentation on the link below:
[mwaa-eks-example](https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html)

The example will create the following resources:
- Install VPC Module
- Install EKS Blueprint Module
- Install Kubernetes Addons Module
- Install aws-mwaa Module
- Create Kubeconfig locally on dags folder
- Create an IAM OIDC provider for the EKS cluster
- Create MWAA namespace
- Create mwaa role and role-binding on EKS
- Add MWAA Service IAM Role on kubeconfig
- Sync Dags folder to the MWAA S3 Bucket
- Sync python requirements.txt file to the MWAA S3 Bucket

The module aws-mwaa included on this example will create the following resources:
- MWAA Environment
- MWAA Service IAM Role and IAM Policy
- MWAA S3 Bucket(To Store Dags, Plugins, Requirements.txt file,etc)
- MWAA Security Group

### Considerations

1. If you used a specific profile when you ran Terraform commands to create the kubeconfig(Line 215 of main.tf) you need to remove the env: section added to the dags/kube_config.yaml file so that it works correctly with Amazon MWAA. To do so, delete the following from the file and then save it:

env:
- name: AWS_PROFILE
value: profile_name

Then you need to run terraform apply again.

2. Ideally we recommend adding the steps to sync requirements/sync dags to the MWAA S3 Bucket as part of a CI/CD pipeline. Generally Dags development have a different lifecycle than the Terraform code to provision infrastructure.
However for simplicity we are providing steps for that using Terraform running AWS CLI commands on null_resource.

## How to Deploy

### Prerequisites:

Ensure that you have installed the following tools in your Mac or Windows Laptop before start working with this module and run Terraform Plan and Apply

1. [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
2. [Kubectl](https://Kubernetes.io/docs/tasks/tools/)
3. [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)

### Deployment Steps

#### Step 1: Clone the repo using the command below

```shell script
git clone https://github.com/aws-ia/terraform-aws-eks-blueprints.git
```

#### Step 2: Run Terraform INIT

Initialize a working directory with configuration files

```shell script
cd examples/mwaa/
terraform init
```

#### Step 3: Run Terraform PLAN

Verify the resources created by this execution

```shell script
export AWS_REGION=<ENTER YOUR REGION> # Select your own region
terraform plan
```

#### Step 4: Finally, Terraform APPLY

to create resources

```shell script
terraform apply
```

Enter `yes` to apply

### Configure `kubectl` and test cluster

EKS Cluster details can be extracted from terraform output or from AWS Console to get the name of cluster.
This following command used to update the `kubeconfig` in your local machine where you run kubectl commands to interact with your EKS Cluster.

#### Step 5: Run `update-kubeconfig` command

`~/.kube/config` file gets updated with cluster details and certificate from the below command

$ aws eks --region <enter-your-region> update-kubeconfig --name <cluster-name>

#### Step 6: List all the worker nodes by running the command below
bryantbiggs marked this conversation as resolved.
Show resolved Hide resolved

$ kubectl get nodes

#### Step 7: List all the pods running in `kube-system` namespace

$ kubectl get pods -n kube-system

#### Step 8: Logging into Apache Airflow

To access your Apache Airflow UI

- Open the Environments page on the Amazon MWAA console.
- Choose an environment.
- Choose Open Airflow UI.

#### Step 9: Trigerring the DAG

Once you are on Airflow UI, enable the example and then trigger it.

![Enable the DAG kubernetes_pod_example ](images/kubernetes_pod_example_dag.png)

![Trigger the DAG kubernetes_pod_example ](images/dag_tree.png)

#### Step 10: Check if the pod was executed successfully

After it runs and completes successfully, use the following command to verify the pod:

```
kubectl get pods -n mwaa
```

You should see output similar to the following:

```
NAME READY STATUS RESTARTS AGE
mwaa-pod-test.4bed823d645844bc8e6899fd858f119d 0/1 Completed 0 25s
```

## How to Destroy

The following command destroys the resources created by `terraform apply`

```shell script
cd examples/mwaa
terraform destroy --auto-approve
```
69 changes: 69 additions & 0 deletions examples/mwaa/aws-mwaa/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# AWS MWAA module
bryantbiggs marked this conversation as resolved.
Show resolved Hide resolved

Terraform module to provision Amazon Managed Workflows for Apache Airflow (MWAA)

## Usage

```terraform
module "mwaa" {
source = "../../modules/aws-mwaa"
environment_name = local.environment_name
airflow_version = local.airflow_version
environment_class = local.environment_class
dag_s3_path = local.dag_s3_path
plugins_s3_path = local.plugins_s3_path
requirements_s3_path = local.requirements_s3_path
logging_configuration = local.logging_configuration
airflow_configuration_options = local.airflow_configuration_options
min_workers = local.airflow_min_workers
max_workers = local.airflow_max_workers
vpc_id = module.aws_vpc.vpc_id
private_subnet_ids = [module.aws_vpc.private_subnets[0], module.aws_vpc.private_subnets[1]]
webserver_access_mode = local.webserver_access_mode
vpn_cidr = local.vpn_cidr
}
```

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.72 |

## Modules

No modules.

## Resources

| Name | Type |
|------|------|
| [aws_kms_alias.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_alias) | resource |
| [aws_kms_key.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_alias"></a> [alias](#input\_alias) | The display name of the alias. The name must start with the word 'alias' followed by a forward slash (alias/) | `string` | n/a | yes |
| <a name="input_deletion_window_in_days"></a> [deletion\_window\_in\_days](#input\_deletion\_window\_in\_days) | The waiting period, specified in number of days. After the waiting period ends, AWS KMS deletes the KMS key. If you specify a value, it must be between 7 and 30, inclusive. If you do not specify a value, it defaults to 30. | `number` | `30` | no |
| <a name="input_description"></a> [description](#input\_description) | The description of the key. | `string` | n/a | yes |
| <a name="input_enable_key_rotation"></a> [enable\_key\_rotation](#input\_enable\_key\_rotation) | Specifies whether annual key rotation is enabled. | `bool` | `true` | no |
| <a name="input_policy"></a> [policy](#input\_policy) | A valid KMS key policy JSON document. Although this is a key policy, not an IAM policy, an aws\_iam\_policy\_document, in the form that designates a principal, can be used. | `string` | n/a | yes |
| <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to the object. | `map(string)` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_key_arn"></a> [key\_arn](#output\_key\_arn) | The Amazon Resource Name (ARN) of the key. |
| <a name="output_key_id"></a> [key\_id](#output\_key\_id) | The globally unique identifier for the key. |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
2 changes: 2 additions & 0 deletions examples/mwaa/aws-mwaa/data.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
Loading