rename control_plane section to remote_access (flyteorg#302)

* rename control_plane section to remote_access - add stub pages for remote access user guide examples - clean up of named tuple outputs example - clean-up of sagemaker distributed pytorch training Signed-off-by: cosmicBboy <[email protected]> * [PR Into 302] Added documentation for running task, launchplans ,inspecting and debgging them (flyteorg#316) * Added documentation for running task, launchplans ,inspecting and debugging them Signed-off-by: Prafulla Mahindrakar <[email protected]> * Incorporated the feedback Signed-off-by: pmahindrakar-oss <[email protected]> Signed-off-by: cosmicBboy <[email protected]> * add links, formatting Signed-off-by: cosmicBboy <[email protected]> Co-authored-by: pmahindrakar-oss <[email protected]>
eapolinario · Jul 21, 2021 · 1049c71 · 1049c71
1 parent 1ecdfd5
commit 1049c71
Show file tree

Hide file tree

Showing 17 changed files with 204 additions and 159 deletions.
diff --git a/cookbook/control_plane/README.rst b/cookbook/control_plane/README.rst
diff --git a/cookbook/control_plane/register_project.py b/cookbook/control_plane/register_project.py
diff --git a/cookbook/control_plane/run_task.py b/cookbook/control_plane/run_task.py
diff --git a/cookbook/control_plane/run_workflow.py b/cookbook/control_plane/run_workflow.py
diff --git a/cookbook/core/flyte_basics/named_outputs.py b/cookbook/core/flyte_basics/named_outputs.py
@@ -1,10 +1,10 @@
 """
-Naming the outputs of a task or a workflow
--------------------------------------------
-As a default, Flyte names the outputs of a task or a workflow using a standardized convention. All outputs are named
-as `o1, o2, o3, ....on` where `o` is the standard prefix and `1,2,..n` is the index position within the return values.
+Named Outputs
+-------------
+By default, Flyte names the outputs of a task or a workflow using a standardized convention. All outputs are named
+as ``o1, o2, o3, ... o<n>.`` where ``o`` is the standard prefix and ``1,2,.. <n>`` is the index position within the return values.
 
-It is also possible to name the outputs of a task or a workflow, so that it may be easier to refer to them, while
+It is also possible to name the outputs of a task or a workflow, so that it's easier to refer to them while
 debugging or visualizing them in the UI. This is not possible to do natively in Python, so flytekit provides an
 alternate way using ``typing.NamedTuple``
 
@@ -20,9 +20,9 @@
 #
 # .. note::
 #
-#  Note the name of the NamedTuple really does not matter. The name of the variable and the type matters. We used a
-#  a default name like `OP`. Also named tuples can be inline, but by convention we prefer to declare them, as pypy
-#   linter errors can be avoided this way
+#  Note the name of the NamedTuple does not matter, but the names and types of the variables do. We used a
+#  a default name like ``OP``. Also named tuples can be inline, but by convention we prefer to declare them, as pypy
+#  linter errors can be avoided this way
 #
 #  .. code-block::
 #
@@ -38,7 +38,7 @@ def say_hello() -> hello_output:
 
 
 # %%
-#  You can also declare the namedtuple ahead of time and then use it in the signature as follows
+# You can also declare the namedtuple ahead of time and then use it in the signature as follows
 wf_outputs = typing.NamedTuple("OP2", greet1=str, greet2=str)
 
 
@@ -47,10 +47,11 @@ def say_hello() -> hello_output:
 # Also as you can see in the workflow, ``say_hello`` returns a tuple, but as with other tuples, you can simply unbox
 # it inline. Also the workflow itself returns a tuple. You can also construct the tuple as you return.
 #
-#  .. note::
+# .. note::
 #
-#       Note that we are de-referencing the individual task execution outputs as for named-outputs uses named-tuples
-#       which are tuples, that need to be de-referenced. 
+#    Note that we are de-referencing the individual task execution outputs as for named-outputs uses named-tuples
+#    which are tuples, that need to be de-referenced.
+
 @workflow
 def my_wf() -> wf_outputs:
     return wf_outputs(say_hello().greet, say_hello().greet)

diff --git a/cookbook/docs/_static/sphx_gallery_autogen.css b/cookbook/docs/_static/sphx_gallery_autogen.css
@@ -4,6 +4,12 @@
 #sphx-glr-download-auto-deployment-deploying-workflows-py,
 #sphx-glr-download-auto-deployment-fast-registration-py,
 #sphx-glr-download-auto-deployment-multiple-k8s-py,
+#sphx-glr-download-auto-remote-access-register-project-py,
+#sphx-glr-download-auto-remote-access-inspecting-executions-py,
+#sphx-glr-download-auto-remote-access-run-launchplan-py,
+#sphx-glr-download-auto-remote-access-run-task-py,
+#sphx-glr-download-auto-remote-access-run-workflow-py,
+#sphx-glr-download-auto-remote-access-debugging-workflows-tasks-py,
 #sphx-glr-download-auto-core-extend-flyte-backend-plugins-py {
     height: 0px;
     visibility: hidden;

diff --git a/cookbook/docs/conf.py b/cookbook/docs/conf.py
@@ -64,6 +64,13 @@ class CustomSorter(FileNameSortKey):
         "use_secrets.py",
         "spot_instances.py",
         "workflow_labels_annotations.py",
+        # Remote Access
+        "register_project.py",
+        "run_task.py",
+        "run_workflow.py",
+        "run_launchplan.py",
+        "inspecting_executions.py",
+        "debugging_workflows_tasks.py",
         # Deployment
         ## Workflow
         "deploying_workflows.py",
@@ -235,7 +242,7 @@ def __call__(self, filename):
     "../testing",
     "../core/containerization",
     "../deployment",
-    # "../control_plane",  # TODO: add content to this section
+    "../remote_access",
     "../integrations/flytekit_plugins/sql",
     "../integrations/flytekit_plugins/papermilltasks",
     "../integrations/flytekit_plugins/pandera",
@@ -261,8 +268,7 @@ def __call__(self, filename):
     "auto/testing",
     "auto/core/containerization",
     "auto/deployment",
-    # "auto/deployment/guides",  # TODO: add content to this section
-    # "auto/control_plane",  # TODO: add content to this section
+    "auto/remote_access",
     "auto/integrations/flytekit_plugins/sql",
     "auto/integrations/flytekit_plugins/papermilltasks",
     "auto/integrations/flytekit_plugins/pandera",

diff --git a/cookbook/docs/index.rst b/cookbook/docs/index.rst
@@ -77,6 +77,15 @@ Table of Contents
 
    ---
 
+   .. link-button:: auto/remote_access/index
+      :type: ref
+      :text: 🎮 Remote Access
+      :classes: btn-block stretched-link
+   ^^^^^^^^^^
+   Register, inspect, and monitor tasks and workflows on a Flyte backend.
+
+   ---
+
    .. link-button:: integrations
       :type: ref
       :text: 🔌  Integrations
@@ -248,6 +257,7 @@ Steps
    Type System <auto/core/type_system/index>
    Testing <auto/testing/index>
    Containerization <auto/core/containerization/index>
+   Remote Access <auto/remote_access/index>
    Configuring Production Features <auto/deployment/index>
    integrations
    Extending flyte <auto/core/extend_flyte/index>

diff --git a/cookbook/integrations/aws/sagemaker_pytorch/sagemaker_pytorch_distributed_training.py b/cookbook/integrations/aws/sagemaker_pytorch/sagemaker_pytorch_distributed_training.py
@@ -2,13 +2,15 @@
 Distributed Pytorch on Sagemaker
 ################################
 
-This example is adapted from the following sagemake example:
-https://github.com/aws/amazon-sagemaker-examples/blob/89831fcf99ea3110f52794db0f6433a4013a5bca/sagemaker-python-sdk/pytorch_mnist/mnist.py
+This example is adapted from
+`this sagemaker example <https://github.com/aws/amazon-sagemaker-examples/blob/89831fcf99ea3110f52794db0f6433a4013a5bca/sagemaker-python-sdk/pytorch_mnist/mnist.py>`__:
 
 It shows how distributed training can be completely performed on the user side with minimal changes using Flyte.
 
-TODO: Flytekit will be adding further simplifications to make writing a distributed training algorithm even simpler, but
-      this example basically provides the full details.
+.. NOTE::
+
+    Flytekit will be adding further simplifications to make writing a distributed training algorithm even simpler, but
+    this example basically provides the full details.
 """
 import logging
 import os

diff --git a/cookbook/remote_access/README.rst b/cookbook/remote_access/README.rst
@@ -0,0 +1,11 @@
+.. _remoteaccess:
+
+Remote Access
+-------------
+
+Flyte provides multiple ways of creating, registering, and inspecting Flyte backend
+entities. The main entities include Flyte tasks, workflows, launchplans, as well as
+their associated execution entities (for more details, see :ref:`divedeep`). This section
+of the user guide covers the different ways that you can programmatically control these
+entities to perform operations like registering tasks, workflows, and launchplans, or
+inspecting and debugging tasks/workflows that are running on a remote backend.
diff --git a/cookbook/control_plane/__init__.py → cookbook/remote_access/__init__.py b/cookbook/control_plane/__init__.py → cookbook/remote_access/__init__.py
diff --git a/cookbook/remote_access/debugging_workflows_tasks.py b/cookbook/remote_access/debugging_workflows_tasks.py
@@ -0,0 +1,28 @@
+"""
+Debugging Workflow and Task Executions
+--------------------------------------
+
+The inspection of task and workflow execution would provide you log links to debug things further
+
+Using ``--details`` flag would shows you node executions view with log links. ::
+
+
+        └── n1 - FAILED - 2021-06-30 08:51:07.3111846 +0000 UTC - 2021-06-30 08:51:17.192852 +0000 UTC
+        │   ├── Attempt :0
+        │       └── Task - FAILED - 2021-06-30 08:51:07.3111846 +0000 UTC - 2021-06-30 08:51:17.192852 +0000 UTC
+        │       └── Logs :
+        │           └── Name :Kubernetes Logs (User)
+        │           └── URI :http://localhost:30082/#/log/flytectldemo-development/f3a5a4034960f4aa1a09-n1-0/pod?namespace=flytectldemo-development
+
+Additionally you can check the pods launched by flyte in <project>-<domain> namespace ::
+
+    kubectl get pods -n <project>-<domain>
+
+The launched pods will have a prefix of execution name along with suffix of nodeId ::
+
+        NAME                        READY   STATUS             RESTARTS   AGE
+        f65009af77f284e50959-n0-0   0/1     ErrImagePull       0          18h
+
+So here the investigation can move ahead by describing the pod and checking the issue with Image pull.
+
+"""
diff --git a/cookbook/remote_access/inspecting_executions.py b/cookbook/remote_access/inspecting_executions.py
@@ -0,0 +1,24 @@
+"""
+Inspecting Workflow and Task Executions
+---------------------------------------
+
+Inspecting workflow and task executions are done in the same manner as below. For more details see the
+`flytectl API reference <https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_get_execution.html>`__.
+
+Monitor the execution by providing the execution id from create command which can be task or workflow execution. ::
+
+    flytectl get execution -p flytesnacks -d development <execid>
+
+For more details use ``--details`` flag which shows node executions along with task executions on them. ::
+
+    flytectl get execution -p flytesnacks -d development <execid> --details
+
+If you prefer to see yaml/json view for the details then change the output format using the -o flag. ::
+
+    flytectl get execution -p flytesnacks -d development <execid> --details -o yaml
+
+To see the results of the execution you can inspect the node closure outputUri in detailed yaml output. ::
+
+    "outputUri": "s3://my-s3-bucket/metadata/propeller/flytesnacks-development-<execid>/n0/data/0/outputs.pb"
+
+"""
diff --git a/cookbook/remote_access/register_project.py b/cookbook/remote_access/register_project.py
@@ -0,0 +1,13 @@
+"""
+Creating a New Project
+-------------------------
+
+Creates project to be used as a home for the flyte resources of tasks and workflows.
+Refer to the `flytectl API reference <https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_project.html>`__
+for more details.
+
+.. prompt:: bash
+
+  flytectl create project --id "my-flyte-project-name" --labels "my-label=my-project-label"  --description "my-flyte-project-name" --name "my-flyte-project-name"
+
+"""
diff --git a/cookbook/remote_access/run_launchplan.py b/cookbook/remote_access/run_launchplan.py
@@ -0,0 +1,28 @@
+"""
+Running a Launchplan
+--------------------
+
+This is multi-steps process where we create an execution spec file, update the spec file and then create the execution.
+More details can be found `here <https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_execution.html>`__.
+
+**Generate an execution spec file** ::
+
+    flytectl get launchplan -p flytesnacks -d development myapp.workflows.example.my_wf  --execFile exec_spec.yaml
+
+**Update the input spec file for arguments to the workflow** ::
+
+    ....
+    inputs:
+        name: "adam"
+    ....
+
+**Create execution using the exec spec file** ::
+
+    flytectl create execution -p flytesnacks -d development --execFile exec_spec.yaml
+
+
+**Monitor the execution by providing the execution id from create command** ::
+
+    flytectl get execution -p flytesnacks -d development <execid>
+
+"""