forked from aws-ia/terraform-aws-eks-blueprints
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add MWAA module and example (aws-ia#585)
Co-authored-by: Bryant Biggs <[email protected]>
- Loading branch information
Showing
11 changed files
with
521 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
## EKS Cluster w/ Amazon Manged Workflows for Apache Airflopw (MWAA) | ||
|
||
The example demonstrates how to use Amazon Managed Workflows for Apache Airflow (MWAA) with Amazon EKS. | ||
|
||
This example was originated from the steps provided on MWAA documentation on the link below: | ||
[mwaa-eks-example](https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html) | ||
|
||
### Considerations | ||
|
||
1. Ideally we recommend adding the steps to sync requirements/sync dags to the MWAA S3 Bucket as part of a CI/CD pipeline. Generally Dags development have a different lifecycle than the Terraform code to provision infrastructure. | ||
However for simplicity we are providing steps for that using Terraform running AWS CLI commands on null_resource. | ||
|
||
## Prerequisites: | ||
|
||
Ensure that you have the following tools installed locally: | ||
|
||
1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) | ||
2. [kubectl](https://Kubernetes.io/docs/tasks/tools/) | ||
3. [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) | ||
|
||
## Deploy | ||
|
||
To provision this example: | ||
|
||
```sh | ||
terraform init | ||
terraform apply | ||
``` | ||
|
||
Enter `yes` at command prompt to apply | ||
|
||
|
||
## Validate | ||
|
||
The following command will update the `kubeconfig` on your local machine and allow you to interact with your EKS Cluster using `kubectl` to validate the deployment. | ||
|
||
1. Run `update-kubeconfig` command: | ||
|
||
```sh | ||
aws eks --region <REGION> update-kubeconfig --name <CLUSTER_NAME> | ||
``` | ||
|
||
2. List the nodes running currently | ||
|
||
```sh | ||
kubectl get nodes | ||
|
||
# Output should look like below | ||
NAME STATUS ROLES AGE VERSION | ||
ip-10-0-30-125.us-west-2.compute.internal Ready <none> 2m19s v1.22.9-eks-810597c | ||
``` | ||
|
||
3. Log into Apache Airflow UI | ||
|
||
- Open the Environments page on the Amazon MWAA console | ||
- Choose an environment | ||
- Under the `Details` section, click the link for the Airflow UI | ||
|
||
4. Triger the DAG workflow to execute | ||
|
||
In the Airflow UI, enable the example and then trigger it. | ||
|
||
![Enable the DAG kubernetes_pod_example ](images/kubernetes_pod_example_dag.png) | ||
|
||
![Trigger the DAG kubernetes_pod_example ](images/dag_tree.png) | ||
|
||
5. Verify that the pod was executed successfully | ||
|
||
After it runs and completes successfully, use the following command to verify the pod: | ||
|
||
```sh | ||
kubectl get pods -n mwaa | ||
``` | ||
|
||
You should see output similar to the following: | ||
|
||
```sh | ||
NAME READY STATUS RESTARTS AGE | ||
mwaa-pod-test.4bed823d645844bc8e6899fd858f119d 0/1 Completed 0 25s | ||
``` | ||
|
||
## Destroy | ||
|
||
To teardown and remove the resources created in this example: | ||
|
||
```sh | ||
terraform destroy -auto-approve | ||
``` |
33 changes: 33 additions & 0 deletions
33
examples/managed-workflow-apache-airflow/dags/mwaa_pod_example.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
from airflow import DAG | ||
from datetime import datetime | ||
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import ( | ||
KubernetesPodOperator, | ||
) | ||
|
||
default_args = { | ||
"owner": "aws", | ||
"depends_on_past": False, | ||
"start_date": datetime(2019, 2, 20), | ||
"provide_context": True, | ||
} | ||
|
||
dag = DAG("kubernetes_pod_example", default_args=default_args, schedule_interval=None) | ||
|
||
# use a kube_config stored in s3 dags folder for now | ||
kube_config_path = "/usr/local/airflow/dags/kube_config.yaml" | ||
|
||
podRun = KubernetesPodOperator( | ||
namespace="mwaa", | ||
image="ubuntu:18.04", | ||
cmds=["bash"], | ||
arguments=["-c", "ls"], | ||
labels={"foo": "bar"}, | ||
name="mwaa-pod-test", | ||
task_id="pod-task", | ||
get_logs=True, | ||
dag=dag, | ||
is_delete_operator_pod=False, | ||
config_file=kube_config_path, | ||
in_cluster=False, | ||
cluster_context="mwaa", # Must match kubeconfig context | ||
) |
2 changes: 2 additions & 0 deletions
2
examples/managed-workflow-apache-airflow/dags/requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
kubernetes==11.0.0 | ||
apache-airflow-providers-cncf-kubernetes==2.0.2 |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+96.3 KB
examples/managed-workflow-apache-airflow/images/kubernetes_pod_example_dag.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.