From d2fd2f797e38fb11225a02319257f22b92a3a7cf Mon Sep 17 00:00:00 2001 From: Samhita Alla Date: Thu, 30 Dec 2021 15:42:12 +0530 Subject: [PATCH 1/2] ideal flow for DevOps Signed-off-by: Samhita Alla --- rsts/deployment/devops.rst | 41 ++++++++++++++++++++++++++++++++++++++ rsts/deployment/index.rst | 9 +++++++++ 2 files changed, 50 insertions(+) create mode 100644 rsts/deployment/devops.rst diff --git a/rsts/deployment/devops.rst b/rsts/deployment/devops.rst new file mode 100644 index 0000000000..8085d9f659 --- /dev/null +++ b/rsts/deployment/devops.rst @@ -0,0 +1,41 @@ +.. _devops-ideal-flow: + +Ideal Flow for DevOps +--------------------- + +Flyte could be useful for many use cases, be it model training, data processing, ELT/ETL, or bioinformatics. +When workflows are built and deployed, be it in any domain, we may need automation to reduce human-in-the-loop to some extent; +essentially, to automate serializing and registering workflows, creating docker containers, etc. + +Before diving into an example use case that explains how DevOps could power Flyte pipelines, let's look at some DevOps-style features Flyte ships with. + +- Every DAG is versioned with Git commit's SHA, which is modifiable +- Tasks and workflows are immutable +- In the case of rapid iteration of code with no dependency modification, :ref:`deployment-fast-registration` can be used +- Multi-tenancy support through projects and domains +- Executions can be monitored using logs on :ref:`Flyte UI ` + +Lyft's Use Case +=============== + +Flyte powers more than 1 million model training and data processing executions per month at Lyft. +DevOps automation has been made part of Flyte pipelines to handle such a massive number of executions. + +- Every time a new PR is created: + + - A Docker container is built from the PR + - Tasks and workflows are registered with Flyte +- If dependencies are modified, PR securely builds the container again +- For just code changes, fast registration is used +- Once a PR is merged, the user simulates a deployment through a deployment pipeline +- At each stage of the deployment, tasks and workflows are registered with a specific domain in Flyte, + where each domain may change the data directory, associated roles, or some meta attributes like labels and annotations +- For production deployments, logs and metrics are automatically tracked, and models are promoted to serving infrastructure +- Users can use interactive notebooks to retrieve intermediate or final outputs, analyze data and automate various monitoring tasks + +Thus, everything is automatically tracked. In fact, multiple users can create isolated PRs and test independently. + +Implementing DevOps when running Flyte pipelines is essential when Flyte is deployed to production. +You could have a team of engineers working on various pipelines with continuous code iteration and deployment. +In such a case, we highly encourage you to incorporate automation to build Flyte pipelines. +This could help speed up your development, iteration, and deployment time! \ No newline at end of file diff --git a/rsts/deployment/index.rst b/rsts/deployment/index.rst index 194548db4b..42c41a7e72 100644 --- a/rsts/deployment/index.rst +++ b/rsts/deployment/index.rst @@ -66,6 +66,14 @@ solution). The following pages will help you effectively deploy and manage an en ^^^^^^^^^^^^ Flyte comes with a lot of things you can configure. These pages will walk you through the various components. + --- + .. link-button:: devops-ideal-flow + :type: ref + :text: Ideal Flow for DevOps + :classes: btn-block stretched-link + ^^^^^^^^^^^^ + An example use case leveraging DevOps practices to run Flyte pipelines. + .. toctree:: :maxdepth: 1 @@ -79,3 +87,4 @@ solution). The following pages will help you effectively deploy and manage an en sandbox plugin_setup/index security/security + devops From 8b7208472ceb88202ed019b4edf657a31b1ebf9b Mon Sep 17 00:00:00 2001 From: Samhita Alla Date: Mon, 3 Jan 2022 10:46:09 +0530 Subject: [PATCH 2/2] incorporate suggestions by @kumare3 Signed-off-by: Samhita Alla --- rsts/deployment/{devops.rst => ideal_flow.rst} | 18 ++++++++++-------- rsts/deployment/index.rst | 6 +++--- 2 files changed, 13 insertions(+), 11 deletions(-) rename rsts/deployment/{devops.rst => ideal_flow.rst} (77%) diff --git a/rsts/deployment/devops.rst b/rsts/deployment/ideal_flow.rst similarity index 77% rename from rsts/deployment/devops.rst rename to rsts/deployment/ideal_flow.rst index 8085d9f659..a30d04914b 100644 --- a/rsts/deployment/devops.rst +++ b/rsts/deployment/ideal_flow.rst @@ -1,7 +1,7 @@ -.. _devops-ideal-flow: +.. _ideal-flow: -Ideal Flow for DevOps ---------------------- +How to Streamline Your Flyte Workflows +-------------------------------------- Flyte could be useful for many use cases, be it model training, data processing, ELT/ETL, or bioinformatics. When workflows are built and deployed, be it in any domain, we may need automation to reduce human-in-the-loop to some extent; @@ -9,14 +9,14 @@ essentially, to automate serializing and registering workflows, creating docker Before diving into an example use case that explains how DevOps could power Flyte pipelines, let's look at some DevOps-style features Flyte ships with. -- Every DAG is versioned with Git commit's SHA, which is modifiable -- Tasks and workflows are immutable +- Every DAG can be versioned with Git commit's SHA or the hash of the code artifact; to know more about versioning, see :ref:`divedeep-versioning` +- Tasks and workflows are immutable, i.e., a version is immutable; any mutation results in a new version - In the case of rapid iteration of code with no dependency modification, :ref:`deployment-fast-registration` can be used - Multi-tenancy support through projects and domains - Executions can be monitored using logs on :ref:`Flyte UI ` -Lyft's Use Case -=============== +Case Study: MLOps at Lyft +========================= Flyte powers more than 1 million model training and data processing executions per month at Lyft. DevOps automation has been made part of Flyte pipelines to handle such a massive number of executions. @@ -27,6 +27,8 @@ DevOps automation has been made part of Flyte pipelines to handle such a massive - Tasks and workflows are registered with Flyte - If dependencies are modified, PR securely builds the container again - For just code changes, fast registration is used + + - Code can be pushed from a local machine where users are not permitted to push Docker containers - Once a PR is merged, the user simulates a deployment through a deployment pipeline - At each stage of the deployment, tasks and workflows are registered with a specific domain in Flyte, where each domain may change the data directory, associated roles, or some meta attributes like labels and annotations @@ -35,7 +37,7 @@ DevOps automation has been made part of Flyte pipelines to handle such a massive Thus, everything is automatically tracked. In fact, multiple users can create isolated PRs and test independently. -Implementing DevOps when running Flyte pipelines is essential when Flyte is deployed to production. +Implementing DevOps when running Flyte pipelines could be essential when Flyte is deployed to production. You could have a team of engineers working on various pipelines with continuous code iteration and deployment. In such a case, we highly encourage you to incorporate automation to build Flyte pipelines. This could help speed up your development, iteration, and deployment time! \ No newline at end of file diff --git a/rsts/deployment/index.rst b/rsts/deployment/index.rst index 42c41a7e72..996ace485a 100644 --- a/rsts/deployment/index.rst +++ b/rsts/deployment/index.rst @@ -67,9 +67,9 @@ solution). The following pages will help you effectively deploy and manage an en Flyte comes with a lot of things you can configure. These pages will walk you through the various components. --- - .. link-button:: devops-ideal-flow + .. link-button:: ideal-flow :type: ref - :text: Ideal Flow for DevOps + :text: How to Streamline Your Flyte Workflows :classes: btn-block stretched-link ^^^^^^^^^^^^ An example use case leveraging DevOps practices to run Flyte pipelines. @@ -87,4 +87,4 @@ solution). The following pages will help you effectively deploy and manage an en sandbox plugin_setup/index security/security - devops + ideal_flow