From b4befef9c0fcdf28bddcdee2c2a617bbd43c5b4e Mon Sep 17 00:00:00 2001 From: James Mishra Date: Thu, 29 Oct 2020 02:28:08 -0500 Subject: [PATCH] Added Prefect as another ETL provider Feel free to change the description for Prefect as needed. I was not sure how much to write. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index e777caf..3d5d568 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ A curated list of notable ETL (extract, transform, load) frameworks, libraries a * [Toil](https://toil.readthedocs.io/en/latest/) - Similar to Luigi, jobs are classes with a run method. Supports executing jobs on other machines (workers) which can include AWS spot instances. * [Argo](https://argoproj.github.io/) - Container based workflow management system for Kubernetes. Workflows are specified as a directed acyclic graph (DAG), and each step is executed on a container, and the latter is run on a Kubernetes Pod. There is also support for Airflow DAGs. * [Dagster](https://dagster.io) - "Dagster is a data orchestrator for machine learning, analytics, and ETL. It lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke." +* [Prefect](https://www.prefect.io/) - Prefect is a relatively new workflow management system for Python. It is similar to Airflow and Luigi. ## Job Scheduling * [Chronos](https://github.com/mesos/chronos) - "a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration."