Airflow DAGS example.

Simple ETL pipeline example. It only has didactic purposes. Not configured for production environment.

Dependencies

• helm (tested with v3.4.1)
• kubectl (tested with v1.18)

Deployments

Types of available deployments:

• Airflow deployment with CeleryExecutor

Use Helm to deploy all Airflow configuration.

Add Airflow chart repository if not installed:

helm repo add airflow-stable https://airflow-helm.github.io/charts and helm repo update

To install the Airflow chart into your Kubernetes cluster:

helm install airflow --namespace airflow airflow-stable/airflow --values [CELERY YAML]

DAGs examples

etl_dag_celery.py : DAG example for CeleryExecutor.

Python packages must be added to requirements.txt.