Skip to content

A repository containing Airflow DAGs that are used for Digital Earth Africa.

License

Notifications You must be signed in to change notification settings

digitalearthafrica/deafrica-airflow

Repository files navigation

DEAfrica Airflow

A repository containing Airflow DAGs that are used for Digital Earth Africa.

Development

Local Editing of DAG's

DAG's can be locally edited and validated. Development can be done in conda or venv according to developer preference. Grab everything airflow and write DAG's. Use autopep8 and pylint to achieve import validation and consistent formatting as the CI pipeline for this repository matures.

pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]
pip install pylint pylint-airflow

pylint dags plugins

Pre-commit setup

Install pip modules

    pip install apache-airflow[aws,kubernetes,postgres,redis,ssh,celery]==1.10.11
    pip install shapely pyproj
    pip install pylint pylint-airflow

A pre-commit config is provided to automatically format and check your code changes. This allows you to immediately catch and fix issues before you raise a failing pull request (which run the same checks under Travis).

If you don't use Conda, install pre-commit from pip:

pip install pre-commit

If you do use Conda, install from conda-forge (required because the pip version uses virtualenvs which are incompatible with Conda's environments)

conda install pre_commit

Now install the pre-commit hook to the current repository:

pre-commit install

Your code will now be formatted and validated before each commit. You can also invoke it manually by running pre-commit run --all-files

Branch Deployment Layout

Airflow Instance Branch
Airflow Oregon develop
Airflow Cape Town Prod main
Airflow Cape Town Dev develop-af

About

A repository containing Airflow DAGs that are used for Digital Earth Africa.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages