diff --git a/.gitignore b/.gitignore index addbc5bb63..819c940142 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,4 @@ dist *.db vendor/ /docker/sandbox-bundled/images/tar +rsts/_tags/ diff --git a/doc-requirements.in b/doc-requirements.in index 51d4018516..321259c61c 100644 --- a/doc-requirements.in +++ b/doc-requirements.in @@ -12,3 +12,8 @@ sphinxcontrib-mermaid sphinxcontrib-video sphinxcontrib-yt sphinx-tabs +sphinx-tags +grpcio<1.49.0 +grpcio-status<1.49.0 +sphinx-tabs<3.4.1 + diff --git a/doc-requirements.txt b/doc-requirements.txt index 87d647b6ca..b8251c69f0 100644 --- a/doc-requirements.txt +++ b/doc-requirements.txt @@ -1,63 +1,94 @@ # -# This file is autogenerated by pip-compile with python 3.8 +# This file is autogenerated by pip-compile with python 3.9 # To update, run: # # pip-compile doc-requirements.in # alabaster==0.7.12 # via sphinx -astroid==2.11.2 +astroid==2.12.12 # via sphinx-autoapi -babel==2.9.1 +babel==2.11.0 # via sphinx beautifulsoup4==4.11.1 # via # furo # sphinx-code-include -certifi==2021.10.8 +certifi==2022.9.24 # via requests -charset-normalizer==2.0.12 +cfgv==3.3.1 + # via pre-commit +charset-normalizer==2.1.1 # via requests +distlib==0.3.6 + # via virtualenv docutils==0.17.1 # via # sphinx # sphinx-panels # sphinx-tabs +filelock==3.8.0 + # via virtualenv furo @ git+https://github.com/flyteorg/furo@main # via -r doc-requirements.in -idna==3.3 +googleapis-common-protos==1.57.0 + # via grpcio-status +grpcio==1.48.2 + # via + # -r doc-requirements.in + # grpcio-status +grpcio-status==1.48.2 + # via -r doc-requirements.in +identify==2.5.8 + # via pre-commit +idna==3.4 # via requests -imagesize==1.3.0 +imagesize==1.4.1 # via sphinx -importlib-metadata==4.11.3 +importlib-metadata==5.0.0 # via sphinx -jinja2==3.1.1 +jinja2==3.0.3 # via # sphinx # sphinx-autoapi -lazy-object-proxy==1.7.1 + # sphinx-tabs +lazy-object-proxy==1.8.0 # via astroid markupsafe==2.1.1 # via jinja2 +nodeenv==1.7.0 + # via pre-commit packaging==21.3 # via sphinx -pbr==5.8.1 +pbr==5.11.0 # via sphinxcontrib-video -pygments==2.11.2 +platformdirs==2.5.4 + # via virtualenv +pre-commit==2.20.0 + # via sphinx-tags +protobuf==4.21.9 + # via + # googleapis-common-protos + # grpcio-status +pygments==2.13.0 # via + # furo # sphinx # sphinx-prompt # sphinx-tabs -pyparsing==3.0.8 +pyparsing==3.0.9 # via packaging -pytz==2022.1 +pytz==2022.6 # via babel pyyaml==6.0 - # via sphinx-autoapi -requests==2.27.1 + # via + # pre-commit + # sphinx-autoapi +requests==2.28.1 # via sphinx six==1.16.0 # via + # grpcio # sphinx-code-include # sphinxext-remoteliteralinclude snowballstemmer==2.2.0 @@ -69,6 +100,7 @@ sphinx==4.5.0 # -r doc-requirements.in # furo # sphinx-autoapi + # sphinx-basic-ng # sphinx-code-include # sphinx-copybutton # sphinx-fontawesome @@ -76,13 +108,15 @@ sphinx==4.5.0 # sphinx-panels # sphinx-prompt # sphinx-tabs + # sphinx-tags # sphinxcontrib-yt - # sphinxext-remoteliteralinclude -sphinx-autoapi==1.8.4 +sphinx-autoapi==2.0.0 # via -r doc-requirements.in +sphinx-basic-ng==1.0.0b1 + # via furo sphinx-code-include==1.1.1 # via -r doc-requirements.in -sphinx-copybutton==0.5.0 +sphinx-copybutton==0.5.1 # via -r doc-requirements.in sphinx-fontawesome==0.0.6 # via -r doc-requirements.in @@ -92,7 +126,9 @@ sphinx-panels==0.6.0 # via -r doc-requirements.in sphinx-prompt==1.5.0 # via -r doc-requirements.in -sphinx-tabs==3.3.1 +sphinx-tabs==3.4.0 + # via -r doc-requirements.in +sphinx-tags==0.1.6 # via -r doc-requirements.in sphinxcontrib-applehelp==1.0.2 # via sphinx @@ -112,17 +148,21 @@ sphinxcontrib-video==0.0.1.dev3 # via -r doc-requirements.in sphinxcontrib-yt==0.2.2 # via -r doc-requirements.in -sphinxext-remoteliteralinclude==0.3.0 +sphinxext-remoteliteralinclude==0.4.0 # via -r doc-requirements.in -typing-extensions==4.1.1 +toml==0.10.2 + # via pre-commit +typing-extensions==4.4.0 # via astroid -unidecode==1.3.4 +unidecode==1.3.6 # via sphinx-autoapi -urllib3==1.26.9 +urllib3==1.26.12 # via requests -wrapt==1.14.0 +virtualenv==20.16.7 + # via pre-commit +wrapt==1.14.1 # via astroid -zipp==3.8.0 +zipp==3.10.0 # via importlib-metadata # The following packages are considered to be unsafe in a requirements file: diff --git a/rsts/community/contribute.rst b/rsts/community/contribute.rst index bc2b895756..9ee93b01b6 100644 --- a/rsts/community/contribute.rst +++ b/rsts/community/contribute.rst @@ -1,10 +1,12 @@ .. _contribute_Flyte: -###################### +##################### Contributing to Flyte -###################### +##################### -Thank you for taking the time to contribute to Flyte! +.. tags:: Contribute, Basic + +Thank you for taking the time to contribute to Flyte! Please read our `Code of Conduct `__ before contributing to Flyte. Here are some guidelines for you to follow, which will make your first and follow-up contributions easier. diff --git a/rsts/community/troubleshoot.rst b/rsts/community/troubleshoot.rst index 8d162c239e..af39507e32 100644 --- a/rsts/community/troubleshoot.rst +++ b/rsts/community/troubleshoot.rst @@ -3,6 +3,8 @@ Troubleshooting Guide --------------------- +.. tags:: Troubleshoot, Basic + .. admonition:: Why did we craft this guide? To help streamline your onboarding experience as much as possible, and sort out common issues. diff --git a/rsts/concepts/admin.rst b/rsts/concepts/admin.rst index 6c8692aab4..80c3f11fbe 100644 --- a/rsts/concepts/admin.rst +++ b/rsts/concepts/admin.rst @@ -1,8 +1,10 @@ .. _divedeep-admin: -########### +########## FlyteAdmin -########### +########## + +.. tags:: Advanced, Design Admin Structure =============== @@ -14,7 +16,6 @@ FlyteAdmin serves as the main Flyte API to process all client requests to the sy Below, we'll dive into each component defined in admin in more detail. - RPC --- diff --git a/rsts/concepts/architecture.rst b/rsts/concepts/architecture.rst index 6dae0b8adc..a278732012 100644 --- a/rsts/concepts/architecture.rst +++ b/rsts/concepts/architecture.rst @@ -4,6 +4,8 @@ Component Architecture ###################### +.. tags:: Advanced, Glossary, Design + This document aims to demystify how Flyte's major components ``Flyteidl``, ``Flytekit``, ``Flytectl``, ``FlyteConsole``, ``FlyteAdmin``, ``FlytePropeller``, and ``FlytePlugins`` fit together at a high level. FlyteIDL diff --git a/rsts/concepts/catalog.rst b/rsts/concepts/catalog.rst index 9c84a70184..8b092e73c0 100644 --- a/rsts/concepts/catalog.rst +++ b/rsts/concepts/catalog.rst @@ -3,6 +3,8 @@ What is Data Catalog? ===================== +.. tags:: Advanced, Design + `DataCatalog `__ is a service to index parameterized, strongly-typed data artifacts across revisions. It allows clients to query artifacts based on meta information and tags. diff --git a/rsts/concepts/component_architecture/flytepropeller_architecture.rst b/rsts/concepts/component_architecture/flytepropeller_architecture.rst index a21b29d5fc..65bb811721 100644 --- a/rsts/concepts/component_architecture/flytepropeller_architecture.rst +++ b/rsts/concepts/component_architecture/flytepropeller_architecture.rst @@ -4,7 +4,9 @@ FlytePropeller Architecture ########################### -.. note:: +.. tags:: Advanced, Design + +.. note:: In the frame of this document, we use the term “workflow” to describe the single execution of a workflow definition. Introduction @@ -13,7 +15,7 @@ Introduction A Flyte :ref:`workflow ` is represented as a Directed Acyclic Graph (DAG) of interconnected Nodes. Flyte supports a robust collection of Node types to ensure diverse functionality. - ``TaskNodes`` support a plugin system to externally add system integrations. - Control flow can be altered during runtime using ``BranchNodes``, which prune downstream evaluation paths based on input. -- ``DynamicNodes`` add nodes to the DAG. +- ``DynamicNodes`` add nodes to the DAG. - ``WorkflowNodes`` allow embedding workflows within each other. FlytePropeller is responsible for scheduling and tracking execution of Flyte workflows. It is implemented using a K8s controller and adheres to the established K8s design principles. In this scheme, resources are periodically evaluated and the goal is to transition from the observed state to a requested state. diff --git a/rsts/concepts/component_architecture/native_scheduler_architecture.rst b/rsts/concepts/component_architecture/native_scheduler_architecture.rst index 2c6d0f6608..24e40557fa 100644 --- a/rsts/concepts/component_architecture/native_scheduler_architecture.rst +++ b/rsts/concepts/component_architecture/native_scheduler_architecture.rst @@ -4,6 +4,8 @@ Flyte Native Scheduler Architecture ################################### +.. tags:: Advanced, Design + Introduction ============ Any workflow engine needs functionality to support scheduled executions. Flyte fulfills this using an in-built native scheduler, which schedules fixed rate and cron-based schedules. The workflow author specifies the schedule during the `launchplan creation `__ and `activates or deactivates `__ the schedule using the `admin APIs `__ exposed for the launch plan. diff --git a/rsts/concepts/console.rst b/rsts/concepts/console.rst index 61eb4bb17d..4cdf9614b1 100644 --- a/rsts/concepts/console.rst +++ b/rsts/concepts/console.rst @@ -1,8 +1,10 @@ .. _divedeep-console: -############# +############ FlyteConsole -############# +############ + +.. tags:: Intermediate, Contribute FlyteConsole is the web UI for the Flyte platform. Here's a video that dives into the graph UX: diff --git a/rsts/concepts/data_management.rst b/rsts/concepts/data_management.rst index 60b273525a..33a7f499a1 100644 --- a/rsts/concepts/data_management.rst +++ b/rsts/concepts/data_management.rst @@ -4,6 +4,8 @@ Understand How Flyte Handles Data ################################# +.. tags:: Basic, Glossary, Design + Types of Data ============= diff --git a/rsts/concepts/domains.rst b/rsts/concepts/domains.rst index da2c22a35f..bb306924dd 100644 --- a/rsts/concepts/domains.rst +++ b/rsts/concepts/domains.rst @@ -3,6 +3,8 @@ Domains ======= +.. tags:: Basic, Glossary + Domains provide an abstraction to isolate resources and feature configuration for different deployment environments. diff --git a/rsts/concepts/dynamic_spec.rst b/rsts/concepts/dynamic_spec.rst index 9621347e5a..4e9e11ad3c 100644 --- a/rsts/concepts/dynamic_spec.rst +++ b/rsts/concepts/dynamic_spec.rst @@ -3,6 +3,8 @@ Dynamic Job Spec ================ +.. tags:: Basic, Design + A dynamic job spec is a subset of the entire workflow spec that defines a set of tasks, workflows, nodes, and output bindings that control how the job should assemble its outputs. This spec is currently only supported as an intermediate step in running Dynamic Tasks. diff --git a/rsts/concepts/execution_timeline.rst b/rsts/concepts/execution_timeline.rst index eae6884d00..276930c94e 100644 --- a/rsts/concepts/execution_timeline.rst +++ b/rsts/concepts/execution_timeline.rst @@ -4,6 +4,8 @@ Timeline of a workflow execution ######################################## +.. tags:: Intermediate, Glossary + The illustration below shows the timeline view of a workflow execution. .. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/deployment/monitoring/flyte_wf_timeline.svg?sanitize=true diff --git a/rsts/concepts/executions.rst b/rsts/concepts/executions.rst index 712693bde1..b6ee602520 100644 --- a/rsts/concepts/executions.rst +++ b/rsts/concepts/executions.rst @@ -3,6 +3,9 @@ ########## Executions ########## + +.. tags:: Basic, Glossary + **Executions** are instances of workflows, nodes or tasks created in the system as a result of a user-requested execution or a scheduled execution. Typical Flow Using Flytectl diff --git a/rsts/concepts/flyte_console.rst b/rsts/concepts/flyte_console.rst index adef153bb1..8e9484789b 100644 --- a/rsts/concepts/flyte_console.rst +++ b/rsts/concepts/flyte_console.rst @@ -3,11 +3,13 @@ How to Use Flyte UI =================== +.. tags:: Basic, UI + Flyte UI is a web-based user interface for Flyte. It helps interact with Flyte objects and builds DAGs out of your workflows. With Flyte UI, you can: -* Launch tasks +* Launch tasks * Launch workflows * View Versioned Tasks and Workflows * Trigger Versioned Tasks and Workflows diff --git a/rsts/concepts/launchplans.rst b/rsts/concepts/launchplans.rst index fe479c738f..a6f86f71a7 100644 --- a/rsts/concepts/launchplans.rst +++ b/rsts/concepts/launchplans.rst @@ -1,7 +1,10 @@ .. _divedeep-launchplans: - + Launch plans -============= +============ + +.. tags:: Basic, Glossary, Design + Launch plans help execute workflows. A workflow can be associated with multiple launch plans and launch plan versions, but an individual launch plan is always associated with a single, specific workflow. After creating a launch plan, it is easy to share and execute them. Launch plans provide a way to templatize Flyte workflow invocations. Launch plans contain a set of bound workflow inputs that are passed as arguments to create an execution. Launch plans do not necessarily contain the entire set of required workflow inputs, but a launch plan is always necessary to trigger an execution. Additional input arguments can be provided at execution time to supplement launch plan static input values. @@ -11,7 +14,8 @@ plan can optionally define a single schedule (which can be easily disabled by di optional notifications. Refer to the :ref:`deployment-cluster-config-notifications` for a deep dive into available notifications. The Association between Workflows and LaunchPlans --------------------------------------------------- +------------------------------------------------- + Every workflow comes with a `default` launch plan that has the same name as that of a workflow. The default launch plan is authored (in code) as part of creating a new workflow. A launch plan version can only ever be mapped to one workflow version; meaning a launch plan version cannot be used twice. This is because part of what makes a new launch plan version is the mapping to the specific workflow version. diff --git a/rsts/concepts/nodes.rst b/rsts/concepts/nodes.rst index b733c4a54f..d67c15457c 100644 --- a/rsts/concepts/nodes.rst +++ b/rsts/concepts/nodes.rst @@ -3,6 +3,8 @@ Nodes ===== +.. tags:: Basic, Glossary + A node represents a unit of execution or work within a workflow. Ordinarily, a node encapsulates an instance of a :ref:`task `, but it can also contain an entire subworkflow or trigger an external workflow. Nodes can have inputs and outputs, which are used to coordinate task inputs and outputs. diff --git a/rsts/concepts/projects.rst b/rsts/concepts/projects.rst index db7073e554..99ed0daf3f 100644 --- a/rsts/concepts/projects.rst +++ b/rsts/concepts/projects.rst @@ -2,6 +2,9 @@ Projects ======== + +.. tags:: Basic, Glossary + A project in Flyte is a group of :ref:`workflows ` and :ref:`tasks ` tied together to achieve a goal. A Flyte project can map to an engineering project or everything that's owned by a team or an individual. There cannot be multiple projects with the same name in Flyte. diff --git a/rsts/concepts/registration.rst b/rsts/concepts/registration.rst index d48e00cd9b..bc745f7a0f 100644 --- a/rsts/concepts/registration.rst +++ b/rsts/concepts/registration.rst @@ -4,6 +4,8 @@ Registration ############ +.. tags:: Basic, Glossary, Design + During registration, Flyte validates the workflow structure and saves the workflow. The registration process also updates the workflow graph. .. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/concepts/executions/flyte_wf_registration_overview.svg?sanitize=true diff --git a/rsts/concepts/schedules.rst b/rsts/concepts/schedules.rst index 664579c00d..6341e1ae19 100644 --- a/rsts/concepts/schedules.rst +++ b/rsts/concepts/schedules.rst @@ -1,7 +1,10 @@ .. _concepts-schedules: Schedules -========== +========= + +.. tags:: Basic, Glossary + Workflows can be run automatically using :ref:`schedules ` associated with launch plans. Only one launch plan version for a given {Project, Domain, Name} combination can be active, which means only one schedule can be active for a launch plan. This is because a single active schedule can exist across all versions of the launch plan. @@ -26,7 +29,7 @@ Cron expression strings use :ref:`this ` syntax. They are .. _rate_unit: Format ---------------- +------ A cron expression represents a set of times, with the help of 5 space-separated fields. @@ -50,13 +53,13 @@ A cron expression represents a set of times, with the help of 5 space-separated Cron schedules ----------------- +-------------- An incorrect cron schedule expression leads to a failure in triggering the schedule. :ref:`Here ` is a table that shows the format of a cron expression. Below is another example: .. code-block:: default - + cron_lp_every_min_of_hour = LaunchPlan.get_or_create( name="my_cron_scheduled_lp", workflow=date_formatter_wf, @@ -66,7 +69,7 @@ Below is another example: schedule="@hourly", # Following schedule runs every hour at beginning of the hour kickoff_time_input_arg="kickoff_time", ), - + ) diff --git a/rsts/concepts/state_machine.rst b/rsts/concepts/state_machine.rst index e8b28f765f..ce570e2f3e 100644 --- a/rsts/concepts/state_machine.rst +++ b/rsts/concepts/state_machine.rst @@ -1,8 +1,10 @@ .. _divedeep-state-machine: -################################################# +################################################ Understanding the State Transition in a Workflow -################################################# +################################################ + +.. tags:: Basic, Design High Level Overview of How a Workflow Progresses to Success =========================================================== diff --git a/rsts/concepts/tasks.rst b/rsts/concepts/tasks.rst index 9bc4772dda..8163c965d9 100644 --- a/rsts/concepts/tasks.rst +++ b/rsts/concepts/tasks.rst @@ -1,9 +1,11 @@ .. _divedeep-tasks: Tasks -====== +===== -Tasks are fully independent units of execution and first-class entities of Flyte. +.. tags:: Basic, Glossary + +Tasks are fully independent units of execution and first-class entities of Flyte. They are the fundamental building blocks and extension points that encapsulate the users' code. Characteristics @@ -34,7 +36,7 @@ When deciding if a unit of execution constitutes a Flyte task, consider these qu Dynamic Tasks -------------- -"Dynamic tasks" is a misnomer. +"Dynamic tasks" is a misnomer. Flyte is one-of-a-kind workflow engine that ships with the concept of truly `Dynamic Workflows `__! Users can generate workflows in reaction to user inputs or computed values at runtime. These executions are evaluated to generate a static graph before execution. @@ -48,7 +50,7 @@ Plugins Flyte exposes an extensible model to express tasks in an execution-independent language. It contains first-class task plugins (for example: `Papermill `__, `Great Expectations `__, and :ref:`more `.) -that execute the Flyte tasks. +that execute the Flyte tasks. Almost any action can be implemented and introduced into Flyte as a "Plugin", which includes: - Tasks that run queries on distributed data warehouses like Redshift, Hive, Snowflake, etc. diff --git a/rsts/concepts/versioning.rst b/rsts/concepts/versioning.rst index 1289296c8c..42df830e6c 100644 --- a/rsts/concepts/versioning.rst +++ b/rsts/concepts/versioning.rst @@ -3,6 +3,8 @@ Versions ======== +.. tags:: Basic, Glossary + One of the most important features and reasons for certain design decisions in Flyte is the need for machine learning and data practitioners to experiment. When users experiment, they do so in isolation and try multiple iterations. Unlike traditional software, the users must conduct multiple experiments concurrently with different environments, algorithms, etc. diff --git a/rsts/concepts/workflow_lifecycle.rst b/rsts/concepts/workflow_lifecycle.rst index 06063902b7..6e4ea2f037 100644 --- a/rsts/concepts/workflow_lifecycle.rst +++ b/rsts/concepts/workflow_lifecycle.rst @@ -4,6 +4,8 @@ Understand the Lifecycle of a Flyte Workflow ################################################################# +.. tags:: Basic, Design + Let's understand how Flyte's plugin machinery works and how information flows from one component to another in Flyte. Under the hood, Flyte relies on a primitive called “Plugins”. Every task that you run on Flyte is powered by a plugin. Some of these plugins are native and guaranteed by Flyte system. These native plugins, for example, run your Flyte tasks inside a k8s pod. There are three native plugins, namely, ``Container``, ``K8sPod``, and ``Sql``. diff --git a/rsts/concepts/workflows.rst b/rsts/concepts/workflows.rst index 71acbb4158..f41078fa63 100644 --- a/rsts/concepts/workflows.rst +++ b/rsts/concepts/workflows.rst @@ -3,6 +3,8 @@ Workflows ========= +.. tags:: Basic, Glossary + A workflow is a directed acyclic graph (DAG) of units of work encapsulated by :ref:`nodes `. Specific instantiations of a workflow (commonly bound with input arguments) are referred to as **workflow executions**, or just executions. In other words, a workflow is a template for an ordered task execution. diff --git a/rsts/conf.py b/rsts/conf.py index 4ea538ef43..6dcf241927 100644 --- a/rsts/conf.py +++ b/rsts/conf.py @@ -23,14 +23,14 @@ # -- Project information ----------------------------------------------------- -project = u"Flyte" -copyright = u"2022, Flyte Authors" -author = u"Flyte" +project = "Flyte" +copyright = "2022, Flyte Authors" +author = "Flyte" # The short X.Y version -version = u"" +version = "" # The full version, including alpha/beta/rc tags -release = u"1.3.0-b2" +release = "1.3.0-b2" # -- General configuration --------------------------------------------------- @@ -64,6 +64,7 @@ "sphinxcontrib.video", "sphinxcontrib.yt", "sphinx_tabs.tabs", + "sphinx_tags", ] extlinks = { @@ -98,7 +99,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path . -exclude_patterns = [u"_build", "Thumbs.db", ".DS_Store"] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # -- Options for HTML output ------------------------------------------------- @@ -181,14 +182,14 @@ # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, "Flyte.tex", u"Flyte Documentation", u"Flyte Authors", "manual"), + (master_doc, "Flyte.tex", "Flyte Documentation", "Flyte Authors", "manual"), ] # -- Options for manual page output ------------------------------------------ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). -man_pages = [(master_doc, "flyte", u"Flyte Documentation", [author], 1)] +man_pages = [(master_doc, "flyte", "Flyte Documentation", [author], 1)] # -- Options for Texinfo output ---------------------------------------------- @@ -199,7 +200,7 @@ ( master_doc, "Flyte", - u"Flyte Documentation", + "Flyte Documentation", author, "Flyte", "Accelerate your ML and data workflows to production.", @@ -211,6 +212,11 @@ autosectionlabel_prefix_document = True autosectionlabel_maxdepth = 2 +# Tags config +tags_create_tags = True +tags_page_title = "Tag" +tags_overview_title = "All Tags" + # -- Options for intersphinx extension --------------------------------------- # Example configuration for intersphinx: refer to the Python standard library. diff --git a/rsts/deployment/aws/index.rst b/rsts/deployment/aws/index.rst index 686fa2805f..c2b636a204 100644 --- a/rsts/deployment/aws/index.rst +++ b/rsts/deployment/aws/index.rst @@ -4,6 +4,8 @@ AWS ###### +.. tags:: AWS, Deployment, Advanced, Infrastructure + .. panels:: :header: text-center :column: col-lg-12 p-2 diff --git a/rsts/deployment/cluster_config/auth_appendix.rst b/rsts/deployment/cluster_config/auth_appendix.rst index bdc6a79256..4759736558 100644 --- a/rsts/deployment/cluster_config/auth_appendix.rst +++ b/rsts/deployment/cluster_config/auth_appendix.rst @@ -4,6 +4,8 @@ Understanding Authentication in Detail ###################################### +.. tags:: Authentication, Design, Advanced + .. _auth-openid-appendix: ************** diff --git a/rsts/deployment/cluster_config/auth_migration.rst b/rsts/deployment/cluster_config/auth_migration.rst index 02799f2215..4af8b064e8 100644 --- a/rsts/deployment/cluster_config/auth_migration.rst +++ b/rsts/deployment/cluster_config/auth_migration.rst @@ -4,6 +4,8 @@ Migrating Your Authentication Config #################################### +.. tags:: Authentication, Infrastructure, Advanced + Flyte previously shipped with only a barebones OIDC setup, and relied on an external authorization server. This migration guide helps you move to Admin's own authorization server. diff --git a/rsts/deployment/cluster_config/auth_setup.rst b/rsts/deployment/cluster_config/auth_setup.rst index 635fd9acea..9e43a96b00 100644 --- a/rsts/deployment/cluster_config/auth_setup.rst +++ b/rsts/deployment/cluster_config/auth_setup.rst @@ -1,8 +1,10 @@ .. _deployment-cluster-config-auth-setup: -######################## +####################### Authenticating in Flyte -######################## +####################### + +.. tags:: Authentication, Infrastructure, Advanced Flyte ships with a canonical implementation of OpenIDConnect client and OAuth2 Server, integrating seamlessly into an organization's existing identity provider. diff --git a/rsts/deployment/cluster_config/cloud_event.rst b/rsts/deployment/cluster_config/cloud_event.rst index 0d65df0a43..ab8b6ce8d0 100644 --- a/rsts/deployment/cluster_config/cloud_event.rst +++ b/rsts/deployment/cluster_config/cloud_event.rst @@ -1,8 +1,10 @@ .. _deployment-cluster-config-cloud-event: -################# +############ Cloud Events -################# +############ + +.. tags:: Infrastructure, AWS, GCP, Advanced Progress of Flyte workflow and task execution is delimited by a series of events that are passed from the FlytePropeller to FlyteAdmin. Administrators @@ -16,6 +18,7 @@ consumption, outside the Flyte platform. ********* Use cases ********* + CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems. @@ -25,6 +28,7 @@ integrating with existing systems within your organization. ************************* Supported Implementations ************************* + Event egress can be configured to work with **AWS** using `SQS `_ and `SNS `_, diff --git a/rsts/deployment/cluster_config/customizable_resources.rst b/rsts/deployment/cluster_config/customizable_resources.rst index de7dad094e..b253b03bdf 100644 --- a/rsts/deployment/cluster_config/customizable_resources.rst +++ b/rsts/deployment/cluster_config/customizable_resources.rst @@ -4,6 +4,8 @@ Adding New Customizable Resources ################################# +.. tags:: Infrastructure, Advanced + As a quick refresher, custom resources allow you to manage configurations for specific combinations of user projects, domains and workflows that override default values. Examples of such resources include execution clusters, task resource defaults, and :std:ref:`more `. @@ -115,7 +117,7 @@ The Flyte plugin registers the resource and the desired quota of every resource The ResourceManager can use a Redis instance as an external store to track and manage resource pool allocation. By default, it is disabled, and can be enabled with: .. code-block:: yaml - + resourcemanager: type: redis resourceMaxQuota: 100 @@ -148,7 +150,7 @@ In this manner, Flyte plugins intelligently throttle resource usage during paral Example ^^^^^^^^ -Let’s take an example to understand resource allocation and deallocation when a plugin requests resources. +Let's take an example to understand resource allocation and deallocation when a plugin requests resources. Flyte has a built-in `Qubole `__ plugin. This plugin allows Flyte tasks to send Hive commands to Qubole. In the plugin, a single Qubole cluster is considered a resource, and sending a single Hive command to a Qubole cluster consumes a token of the corresponding resource. The resource is allocated when the status is **“AllocationGranted”**. Qubole plugin calls: @@ -186,6 +188,6 @@ How can you force ResourceManager to force runtime quota allocation constraints? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Runtime quota allocation constraints can be achieved using ResourceConstraintsSpec. It is a contact that a plugin can specify at different project and namespace levels. -Let’s take an example to understand it. +Let's take an example to understand it. You can set ResourceConstraintsSpec to ``nil`` objects, which means there would be no allocation constraints at the respective project and namespace level. When ResourceConstraintsSpec specifies ``nil`` ProjectScopeResourceConstraint, and a non-nil NamespaceScopeResourceConstraint, it suggests no constraints specified at any project or namespace level. diff --git a/rsts/deployment/cluster_config/datacatalog_config.rst b/rsts/deployment/cluster_config/datacatalog_config.rst index 51275e8ac0..527a909447 100644 --- a/rsts/deployment/cluster_config/datacatalog_config.rst +++ b/rsts/deployment/cluster_config/datacatalog_config.rst @@ -4,6 +4,8 @@ Flyte Datacatalog Configuration ######################################### +.. tags:: Configuration, Advanced + - `application <#section-application>`_ - `database <#section-database>`_ diff --git a/rsts/deployment/cluster_config/eventing.rst b/rsts/deployment/cluster_config/eventing.rst index 8379416264..112fedb5ca 100644 --- a/rsts/deployment/cluster_config/eventing.rst +++ b/rsts/deployment/cluster_config/eventing.rst @@ -1,10 +1,13 @@ .. _deployment-cluster-config-eventing: -################# +############### Platform Events -################# +############### -Progress of Flyte workflow and task execution is delimited by a series of events that are passed from the FlytePropeller to FlyteAdmin. Administrators can configure FlyteAdmin to send these events onwards to a pub/sub system like SNS/SQS as well. Note that this configuration is distinct from the configuration for notifications :ref:`deployment-cluster-config-notifications`. They should use separate topics/queues. These events are meant for external consumption, outside the Flyte platform, whereas the notifications pub/sub setup is entirely for Admin itself to send email/pagerduty/etc notifications. +.. tags:: Configuration, Infrastructure, Advanced + +Progress of Flyte workflow and task execution is delimited by a series of events that are passed from the FlytePropeller to FlyteAdmin. +Administrators can configure FlyteAdmin to send these events onwards to a pub/sub system like SNS/SQS as well. Note that this configuration is distinct from the configuration for notifications :ref:`deployment-cluster-config-notifications`. They should use separate topics/queues. These events are meant for external consumption, outside the Flyte platform, whereas the notifications pub/sub setup is entirely for Admin itself to send email/pagerduty/etc notifications. ********* Use cases @@ -15,6 +18,7 @@ The external events flow can be useful for tracking data lineage and integrating ************************* Supported Implementations ************************* + Event egress can be configured to work with **AWS** using `SQS `_ and `SNS `_ or **GCP** `Cloud Pub/Sub `_. ************* @@ -37,8 +41,10 @@ To turn on, add the following to your FlyteAdmin: type: aws Helm -====== -There should already be a section for this in the ``values.yaml`` file. Update the settings under the ``external_events`` key and turn ``enable`` to ``true``. The same flag is used for Helm as for Admin itself. +==== + +There should already be a section for this in the ``values.yaml`` file. +Update the settings under the ``external_events`` key and turn ``enable`` to ``true``. The same flag is used for Helm as for Admin itself. ***** Usage diff --git a/rsts/deployment/cluster_config/flyteadmin_config.rst b/rsts/deployment/cluster_config/flyteadmin_config.rst index ec79602a79..68b72e9c8d 100644 --- a/rsts/deployment/cluster_config/flyteadmin_config.rst +++ b/rsts/deployment/cluster_config/flyteadmin_config.rst @@ -4,6 +4,8 @@ Flyte Admin Configuration ######################################### +.. tags:: Configuration, Advanced + - `admin <#section-admin>`_ - `auth <#section-auth>`_ diff --git a/rsts/deployment/cluster_config/flytepropeller_config.rst b/rsts/deployment/cluster_config/flytepropeller_config.rst index d57b905940..d461f0f013 100644 --- a/rsts/deployment/cluster_config/flytepropeller_config.rst +++ b/rsts/deployment/cluster_config/flytepropeller_config.rst @@ -4,6 +4,8 @@ Flyte Propeller Configuration ######################################### +.. tags:: Configuration, Advanced + - `admin <#section-admin>`_ - `catalog-cache <#section-catalog-cache>`_ diff --git a/rsts/deployment/cluster_config/general.rst b/rsts/deployment/cluster_config/general.rst index f91544c915..317e85563c 100644 --- a/rsts/deployment/cluster_config/general.rst +++ b/rsts/deployment/cluster_config/general.rst @@ -1,7 +1,9 @@ .. _deployment-cluster-config-general: Configuring Custom K8s Resources ----------------------------------- +-------------------------------- + +.. tags:: Infrastructure, Kubernetes, Advanced *************************** Configurable Resource Types diff --git a/rsts/deployment/cluster_config/monitoring.rst b/rsts/deployment/cluster_config/monitoring.rst index 331a984e87..2214b2c8c1 100644 --- a/rsts/deployment/cluster_config/monitoring.rst +++ b/rsts/deployment/cluster_config/monitoring.rst @@ -3,6 +3,8 @@ Monitoring ---------- +.. tags:: Infrastructure, Advanced + .. tip:: The Flyte core team publishes and maintains Grafana dashboards built using Prometheus data sources, which can be found `here `__. Metrics for Executions diff --git a/rsts/deployment/cluster_config/notifications.rst b/rsts/deployment/cluster_config/notifications.rst index 2bead7358e..efdd452dae 100644 --- a/rsts/deployment/cluster_config/notifications.rst +++ b/rsts/deployment/cluster_config/notifications.rst @@ -3,6 +3,8 @@ Notifications ------------- +.. tags:: Infrastructure, Advanced + When a workflow completes, users can be notified by: * Email diff --git a/rsts/deployment/cluster_config/performance.rst b/rsts/deployment/cluster_config/performance.rst index f983279855..a93ea58628 100644 --- a/rsts/deployment/cluster_config/performance.rst +++ b/rsts/deployment/cluster_config/performance.rst @@ -4,11 +4,13 @@ Optimizing Performance ###################################################### +.. tags:: Infrastructure, Kubernetes, Advanced + .. tip:: Before getting started, it is always important to measure the performance. Flyte project publishes and manages some grafana templates as described in - :ref:`deployment-cluster-config-monitoring`. The video below contains an overview of the Flyte architecture, what is meant by "performance", details of one loop in FlytePropeller, and a demo of the Grafana Labs dashboard. -.. youtube:: FJ-rG9lZDhY +.. youtube:: FJ-rG9lZDhY Scaling up FlytePropeller ========================== diff --git a/rsts/deployment/cluster_config/scheduler_config.rst b/rsts/deployment/cluster_config/scheduler_config.rst index 9744937689..894f5a43ac 100644 --- a/rsts/deployment/cluster_config/scheduler_config.rst +++ b/rsts/deployment/cluster_config/scheduler_config.rst @@ -4,6 +4,8 @@ Flyte Scheduler Configuration ######################################### +.. tags:: Configuration, Advanced + - `admin <#section-admin>`_ - `auth <#section-auth>`_ diff --git a/rsts/deployment/gcp/index.rst b/rsts/deployment/gcp/index.rst index c4f98039a6..7e7fd49d31 100644 --- a/rsts/deployment/gcp/index.rst +++ b/rsts/deployment/gcp/index.rst @@ -4,6 +4,8 @@ GCP ###### +.. tags:: Infrastructure, GCP, Deployment, Advanced + .. panels:: :header: text-center :column: col-lg-12 p-2 diff --git a/rsts/deployment/ideal_flow.rst b/rsts/deployment/ideal_flow.rst index dcbd00b7e3..243d9b1536 100644 --- a/rsts/deployment/ideal_flow.rst +++ b/rsts/deployment/ideal_flow.rst @@ -1,7 +1,9 @@ .. _ideal-flow: Streamlining Your Flyte Workflows --------------------------------------- +--------------------------------- + +.. tags:: Deployment, Basic Flyte has a wide range of applications, including model training, data processing, ELT/ETL, and bioinformatics. Regardless of the domain, when workflows are built and deployed, we may need automation to reduce human-in-the-loop to some extent; @@ -16,7 +18,7 @@ Before diving into an example use case that explains how DevOps could power Flyt - Executions can be monitored using logs on :ref:`Flyte UI `. .. note:: - + Currently, Flyte does not support the promotion of workflows across various stages (development, staging, production). This promotion process is expected to be handled externally through CI/CD process. Case Study: MLOps at Lyft diff --git a/rsts/deployment/multicluster.rst b/rsts/deployment/multicluster.rst index 92876ff31a..c94e1f659b 100644 --- a/rsts/deployment/multicluster.rst +++ b/rsts/deployment/multicluster.rst @@ -5,6 +5,8 @@ Using Multiple Kubernetes Clusters ################################## +.. tags:: Kubernetes, Infrastructure, Advanced + Scaling Beyond Kubernetes ------------------------- diff --git a/rsts/deployment/overview.rst b/rsts/deployment/overview.rst index 7439f5c2e9..ff14323647 100644 --- a/rsts/deployment/overview.rst +++ b/rsts/deployment/overview.rst @@ -4,6 +4,8 @@ Deployment Overview ################### +.. tags:: Deployment, Infrastructure, Advanced + Up until now, the Flyte backend you've been working with has likely been accessible only on ``localhost`` and likely entirely in one Docker container. In order to handle the production load and make use of all the additional features Flyte offers, you need to replace, add, and configure certain components. This page describes at a high-level what a production-ready deployment might look like. ******************* diff --git a/rsts/deployment/plugin_setup/aws/index.rst b/rsts/deployment/plugin_setup/aws/index.rst index 1ead3cf337..1f366690a1 100644 --- a/rsts/deployment/plugin_setup/aws/index.rst +++ b/rsts/deployment/plugin_setup/aws/index.rst @@ -4,6 +4,7 @@ AWS Plugins Setup ################# +.. tags:: AWS, Integration, MachineLearning, Data, Advanced .. panels:: :header: text-center diff --git a/rsts/deployment/plugin_setup/gcp/index.rst b/rsts/deployment/plugin_setup/gcp/index.rst index b61c51b541..f4f00fa907 100644 --- a/rsts/deployment/plugin_setup/gcp/index.rst +++ b/rsts/deployment/plugin_setup/gcp/index.rst @@ -4,6 +4,7 @@ GCP Plugins Setup ################# +.. tags:: GCP, Integration, Data, Advanced .. panels:: :header: text-center diff --git a/rsts/deployment/plugin_setup/k8s/index.rst b/rsts/deployment/plugin_setup/k8s/index.rst index 93ca2c977a..d3f5cc358e 100644 --- a/rsts/deployment/plugin_setup/k8s/index.rst +++ b/rsts/deployment/plugin_setup/k8s/index.rst @@ -2,7 +2,9 @@ K8s Operator ------------------------------------------ +------------ + +.. tags:: Kubernetes, Integration, KubernetesOperator, Spark, AWS, GCP, MachineLearning, DistributedComputing, Advanced This guide gives an overview of setting up the K8s Operator backend plugin in your Flyte deployment. diff --git a/rsts/deployment/plugin_setup/webapi/index.rst b/rsts/deployment/plugin_setup/webapi/index.rst index e80ca21466..7ade2be7b8 100644 --- a/rsts/deployment/plugin_setup/webapi/index.rst +++ b/rsts/deployment/plugin_setup/webapi/index.rst @@ -4,6 +4,7 @@ Web API Plugin Setup #################### +.. tags:: WebAPI, Integration, Data, Advanced .. panels:: :header: text-center diff --git a/rsts/deployment/sandbox.rst b/rsts/deployment/sandbox.rst index d63d720443..f1d918037a 100644 --- a/rsts/deployment/sandbox.rst +++ b/rsts/deployment/sandbox.rst @@ -4,6 +4,8 @@ Sandbox Deployment ################### +.. tags:: Deployment, Infrastructure, Design, Intermediate + .. warning:: The sandbox deployment is not suitable for production environments. For an in-depth overview of how to productionize your Flyte deployment, checkout the :ref:`deployment` guide. diff --git a/rsts/deployment/security/security.rst b/rsts/deployment/security/security.rst index 646945b150..945e0afb3c 100644 --- a/rsts/deployment/security/security.rst +++ b/rsts/deployment/security/security.rst @@ -4,6 +4,8 @@ Security Overview ################### +.. tags:: Kubernetes, Infrastructure, Advanced + Here we cover the security aspects of running your flyte deployments. In the current state, we will cover the user used for running the flyte services, and go through why we do this and not run them as a root user. diff --git a/rsts/index.rst b/rsts/index.rst index 8e8cdfc83a..1b77dd718a 100644 --- a/rsts/index.rst +++ b/rsts/index.rst @@ -54,10 +54,12 @@ API Reference - + What is Flyte? ============== +`Flyte Tags <_tags/tagsindex.html>`__ + .. raw:: html

The workflow automation platform for complex, mission-critical data and machine learning processes at scale.

diff --git a/rsts/reference/swagger.rst b/rsts/reference/swagger.rst index 954af49c87..91d970ef07 100644 --- a/rsts/reference/swagger.rst +++ b/rsts/reference/swagger.rst @@ -4,6 +4,8 @@ Flyte API Playground: Swagger ############################# +.. tags:: Basic + Flyte services expose gRPC services for efficient/low latency communication across all services as well as for external clients (FlyteCTL, FlyteConsole, Flytekit Remote, etc.). The service definitions are defined `here `__.