Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update clis.rst #1124

Merged
merged 5 commits into from
Aug 13, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update clis.rst
Changes based on comments
Signed-off-by: SmritiSatyanV smriti@union.ai
SmritiSatyanV committed Aug 10, 2022
commit 0f4fd449260252d75f437d78d03271f5aba4e6ae
59 changes: 36 additions & 23 deletions docs/source/design/clis.rst
Original file line number Diff line number Diff line change
@@ -23,37 +23,38 @@ Command Line Interfaces
***********************

Flytectl
===========
``flytectl`` is the general CLI that can be used to communicate with the Flyte control plane (FlyteAdmin). It ships with flytekit as part of the Pypi package. Think of this as the ``kubectl`` for Flyte.
Think of this CLI as a network-aware (can talk to Admin) but not code-aware (doesn't need to have user code checked out) CLI. In the registration flow, this CLI is responsible for shipping the compiled Protobuf files off to Flyte Admin.
=========

``flytectl`` is the general CLI to communicate with the Flyte control plane (FlyteAdmin). It ships with flytekit as part of the Pypi package. Think of this as the ``kubectl`` for Flyte.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missed this the first time. flytectl does not ship with flytekit. it needs to be installed separately. Can we link to flytectl docs - https://docs.flyte.org/projects/flytectl/en/latest/


Think of this as a network-aware (can talk to admin) but not code-aware (no need to have user code checked out) CLI. In the registration flow, this CLI is responsible for shipping the compiled Protobuf files off to Flyte Admin.

Pyflyte
=========
Unlike flytectl, think of this CLI as code-aware, but not network-aware (the latter is not entirely true, but it's helpful to think of it that way).
========

This CLI is responsible for the serialization (compilation) step in the registration flow. It will parse through the user code, looking for tasks, workflows and launch plans, and compile them to protobuf files.
Unlike flytectl, think of this CLI as code-aware, which is responsible for the serialization (compilation) step in the registration flow. It will parse through the user code, looking for tasks, workflows, and launch plans, and compile them to protobuf files.

What is ``pyflyte register``?
=============================
``pyflyte register`` is a command that registers all the workflows present in the repository/directory using fast-registration. It is equivalent to using two commands (``pyflyte package`` and ``flytectl register``) to perform the same operation (registration).
The ``pyflyte package`` command parses and compiles the user’s Python code into Flyte protobuf objects.
The ``flytectl register`` command ships the protobuf objects over the network to Flyte control plane, wherein ``flytectl`` also allows adding attributes such as IAM role, K8s service account, and so on.

Why should you use ``pyflyte register``?
========================================
The ``pyflyte register`` command bridges the gap between ``pyflyte package`` + ``flytectl register``and ``pyflyte run`` commands. It offers the functionality of the ``pyflyte package`` (with smarter naming semantics and combines the network call into one step), by registering all the workflows in the repository, and the convenience of the ``pyflyte run``, by launching the execution of a single workflow.
``pyflyte register`` is a command that registers all the workflows present in the repository/directory using fast-registration. It is equivalent to using two commands (``pyflyte package`` and ``flytectl register``) to perform the same operation (registration).

.. note ::
The ``pyflyte package`` command parses and compiles the user’s Python code into Flyte protobuf objects. These compiled objects are not stored, which makes it difficult to introspect the objects.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no they are stored, they're available locally after you run pyflyte package. In pyflyte register, they are not stored.


You can’t use ``pyflyte register`` if you are unaware of the run-time options yet(IAM role, service account, and so on).
The ``flytectl register`` command ships the protobuf objects over the network to the Flyte control plane, wherein ``flytectl`` also allows adding attributes such as IAM role, K8s service account, and so on.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The ``flytectl register`` command ships the protobuf objects over the network to the Flyte control plane, wherein ``flytectl`` also allows adding attributes such as IAM role, K8s service account, and so on.
The ``flytectl register`` command then ships the protobuf objects over the network to the Flyte control plane. In the process, ``flytectl`` also allows the user to set run-time attributes such as IAM role, K8s service account, etc.


What is ``pyflyte run``?
What is ``pyflyte run``?
========================
The ``pyflyte run`` command is a light-weight, convenience command that combines packaging, registering and launching a workflow in a single command.
It is not a fully featured production scale mode of operation, because it
is designed to be a quick and easy iteration tool to get started with Flyte or test small self-contained scripts. The caveat here is it operates on a single file, and this file will have to contain all the required Flyte entities. Let’s take an example so that you can understand it better.
Suppose you execute a script that defines 10 tasks and a workflow that calls only 2 out of the 10 tasks, the remaining 8 tasks don’t get registered at that point.
It is considered fast registration because when a script is executed using ``pyflyte run``, the script is zipped up and sent to FlyteAdmin. When the task is executed in the backend, this zipped file is extracted and used.

The ``pyflyte run`` command is a light-weight, convenience command that incorporates packaging, registering, and launching a workflow into a single command.

It is not a fully featured production scale mode of operation, because it is designed to be a quick and easy iteration tool to get started with Flyte or test small self-contained scripts. The caveat here is it operates on a single file, and this file will have to contain all the required Flyte entities. Let’s take an example so that you can understand it better.

Suppose you execute a script that defines 10 tasks and a workflow that calls only 2 out of the 10 tasks. The remaining 8 tasks don’t get registered at that point.

It is considered fast registration because when a script is executed using ``pyflyte run``, the script is bundled up and uploaded to FlyteAdmin. When the task is executed in the backend, this zipped file is extracted and used.

``pyflyte run`` produces a **portable** package (a .tgz file) of Flyte entities, which helps introspect them at a later time too. You can register this package with multiple Flyte backends. You can save this package, use it for audit, register with a different admin, and so on.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No pyflyte run does not do this. this is pyflyte package.


Usage
=====
@@ -66,10 +67,22 @@ Usage

Neither ``pyflyte register`` nor ``pyflyte run`` commands work on Python namespace packages since both the tools traverse the filesystem to find the first folder that doesn't have an __init__.py file, which is interpreted as the root of the project. Both the commands use this root as the basis to name the Flyte entities.

In a broad way, ``pyflyte register`` is equivalent to ``pyflyte run`` minus launching workflow with the exception that ``pyflyte run`` can only register a single workflow whereas ``pyflyte register`` can register all workflows in a repository.
In a broad way, ``pyflyte register`` is equivalent to ``pyflyte run`` minus launching workflows, with the exception that ``pyflyte run`` can only register a single workflow, whereas ``pyflyte register`` can register all workflows in a repository.

Why should you use ``pyflyte register``?
========================================

The ``pyflyte register`` command bridges the gap between ``pyflyte package`` + ``flytectl register``and ``pyflyte run`` commands. It offers the functionality of the ``pyflyte package`` (with smarter naming semantics and combines the network call into one step), by only registering all the workflows in the repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The ``pyflyte register`` command bridges the gap between ``pyflyte package`` + ``flytectl register``and ``pyflyte run`` commands. It offers the functionality of the ``pyflyte package`` (with smarter naming semantics and combines the network call into one step), by only registering all the workflows in the repository.
The ``pyflyte register`` command bridges the gap between ``pyflyte package`` + ``flytectl register`` and ``pyflyte run`` commands. It offers the functionality of the ``pyflyte package`` (with smarter naming semantics and combines the network call into one step).


.. note ::

You can’t use ``pyflyte register`` if you are unaware of the run-time options yet(IAM role, service account, and so on).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can’t use ``pyflyte register`` if you are unaware of the run-time options yet(IAM role, service account, and so on).
You can’t use ``pyflyte register`` if you are unaware of the run-time options yet (IAM role, service account, and so on).


Should I use ``pyflyte run`` or ``pyflyte package`` + ``flytectl register``?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also differentiate between "pyflyte package + flytectl register" and just "pyflyte register"?

To me there is a difference. The first option will produce a "portable" package of your flyte entities. that is you can take that .tgz file, and register it with ten different flyte backends. you can email it to your friend at a different company and ask them to register it with their admin. you can also save the asset, keep it around for audit, etc.
pyflyte register works well for single admin use-cases, and the compiled objects are lost afterwards, so you can't as easily introspect. In a production Flyte setting, I think each has their place. If I were running something in CI, I would opt for the first route. If i were a user iterating locally, I'd probably opt for the second way. @kumare3 thoughts?

============================================================================
``pyflyte run`` is convenient when you are starting out with Flyte, and testing small scripts.
``pyflyte package`` + ``flytectl register`` is convenient when you want to zip your files (that constitutes all your tasks and workflows) into a single portable file and pass it on to another developer to register it.

Both the commands have their own place in a production Flyte setting.

``pyflyte run`` is convenient when you are starting out with Flyte, testing small scripts, or iterating over local scripts.

``pyflyte package`` + ``flytectl register`` is useful for single admin use-cases where you want to zip your files (that constitute all your tasks and workflows) into a single portable file and pass it on to another developer/location to register it or run something in CI.