Skip to content

Latest commit

 

History

History
28 lines (16 loc) · 802 Bytes

README.md

File metadata and controls

28 lines (16 loc) · 802 Bytes
Raw

Airflow DAGS example.

Simple ETL pipeline example. It only has didactic purposes. Not configured for production environment.

Dependencies

  • helm (tested with v3.4.1)
  • kubectl (tested with v1.18)

Deployments

Types of available deployments:

  • Airflow deployment with CeleryExecutor

Use Helm to deploy all Airflow configuration.

Add Airflow chart repository if not installed:

helm repo add airflow-stable https://airflow-helm.github.io/charts and helm repo update

To install the Airflow chart into your Kubernetes cluster:

helm install airflow --namespace airflow airflow-stable/airflow --values [CELERY YAML]

DAGs examples

etl_dag_celery.py : DAG example for CeleryExecutor.

Python packages must be added to requirements.txt.