From 53187e45900f838d4eb6307ba669de834a192dc3 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 29 Jan 2021 12:23:10 -0800 Subject: [PATCH 1/7] revamp --- rsts/user/getting_started/create_first.rst | 97 ++++++++-------------- 1 file changed, 33 insertions(+), 64 deletions(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index 2ae9b8129f..d5113e7384 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -14,18 +14,15 @@ You can save some effort by cloning the ``flytesnacks`` repo, and re-initializin git init cd python -now open the "Makefile" and change the first line to ``IMAGE_NAME=myflyteproject`` -Let's also remove the existing python task so we can write one from scratch. :: - - rm single_step/edges.py +Let's take a look at how easy it is to use Flyte by writing a task from scratch. Creating a Project ****************** In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project. -Lets create a new project called ``myflyteproject``. Use the project creation endpoint to create the new project :: +For example, to create a new project called ``myflyteproject``: use the project creation endpoint to create the new project :: curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "myflyteproject", "name": "myflyteproject"} }' @@ -37,53 +34,44 @@ The most basic Flyte primitive is a "task". Flyte Tasks are units of work that c Start by creating a new file :: - mkdir -p workflows - touch workflows/first.py -This directory has been marked in the `configuration file `_ as the location to look for workflows and tasks. Begin by importing some of the libraries that we'll need for this example. + touch cookbook/recipes/core/first.py + .. code-block:: python - from __future__ import absolute_import - from __future__ import division - from __future__ import print_function - - import urllib.request as _request + from urllib import request import cv2 - from flytekit.common import utils - from flytekit.sdk.tasks import python_task, outputs, inputs - from flytekit.sdk.types import Types - from flytekit.sdk.workflow import workflow_class, Output, Input + import flytekit + from flytekit import task, workflow + from flytekit.types.file import FlyteFile -From there, we can begin to write our first task. It should look something like this. +From there, we can begin to write our first task. It should look something like this: .. code-block:: python - @inputs(image_location=Types.String) - @outputs(parsed_image=Types.Blob) - @python_task - def edge_detection_canny(wf_params, image_location, parsed_image): - with utils.AutoDeletingTempDir('test') as tmpdir: - plane_fname = '{}/plane.jpg'.format(tmpdir.name) - with _request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile: - data = d.read() - opfile.write(data) - - img = cv2.imread(plane_fname, 0) - edges = cv2.Canny(img, 50, 200) # hysteresis thresholds - - output_file = '{}/output.jpg'.format(tmpdir.name) - cv2.imwrite(output_file, edges) - - parsed_image.set(output_file) + @task + def edge_detection_canny(image_location:str) -> FlyteFile: + working_dir = flytekit.current_context().working_directory: + plane_fname = '{}/plane.jpg'.format(working_dir.name) + with request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile: + data = d.read() + opfile.write(data) + + img = cv2.imread(plane_fname, 0) + edges = cv2.Canny(img, 50, 200) # hysteresis thresholds + + output_file = '{}/output.jpg'.format(working_dir.name) + cv2.imwrite(output_file, edges) + + return FlyteFile["jpg"](path=output_file) Some of the new concepts demonstrated here are: -* ``wf_params``: The first argument to a python task is a Flyte SDK defined object that offers handlers like logging. -* Inputs and outputs are first defined in the decorator, and then passed into the argument of the function. Note that the names in the function signature need to match those in the decorator arguments. -* A ``Blob`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. +* Use the ``@task`` decorator to convert your typed python function to a Flyte task. +* A ``FlyteFile`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. Writing a Workflow ********************* @@ -91,35 +79,16 @@ Next you need to call that task from a workflow. In the same file, add these li .. code-block:: python - @workflow_class - class EdgeDetectorWf(object): - image_input = Input(Types.String, required=True, help="Image to run for") - run_edge_detection = edge_detection_canny(image_location=image_input) - edges = Output(run_edge_detection.outputs.parsed_image, sdk_type=Types.Blob) - + @workflow + class EdgeDetectorWf(image_input: str) -> FlyteFile: + edges = edge_detection_canny(image_location=image_input) + return edges + This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image. +You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate locally before moving on to register it with Flyte. Interacting with Flyte ************************ -Flyte fulfills tasks using docker images. You'll need to build a docker image from this code before it can run in Flyte. The repo has a make target to build the docker image for you :: - - eval $(minikube -p minikube docker-env) # If using the flyte sandbox via minikube - make docker_build - -If you have the flyte sandbox installed on your local machine, the image will be accessible to your Flyte system. If you're running a remote Flyte instance, you'll need to upload this image to a remote registry such as Dockerhub, Amazon ECR, or Google Container Registry, so that it can be used by the Flyte system. - -To upload to a remote registry (or even local registry), use :: - - DOCKER_REGISTRY_USERNAME={username} DOCKER_REGISTRY_PASSWORD={pass} REGISTRY=docker.io make docker_build - -Replace the values above with your registry username, password, and registry endpoint. - -You may need to change the ``IMAGE_NAME`` in the Makefile to reflect your namespace in the docker registry. (ex ``{{my docker username}}/myflyteproject``) - -With the image built, we just need to register the tasks and workflows. The process is the same as what we had done previously. :: - - docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' {{ your docker image }} pyflyte -p myflyteproject -d development -c sandbox.config register workflows - -After this, you should be able to visit the Flyte UI, and run the workflow as you did with ``flytesnacks`` previously. +For detailed and interactive example workflow 'recipes' check out the `Flytesnacks Cookbook `_ From 4f2991594cf01f496e302f9cee34c262c6fa2c4f Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 29 Jan 2021 12:31:58 -0800 Subject: [PATCH 2/7] Update rsts/user/getting_started/create_first.rst Co-authored-by: Yee Hing Tong --- rsts/user/getting_started/create_first.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index d5113e7384..e7b15e4282 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -91,4 +91,4 @@ You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate local Interacting with Flyte ************************ -For detailed and interactive example workflow 'recipes' check out the `Flytesnacks Cookbook `_ +For detailed and interactive example workflow 'recipes' check out the `Flytesnacks Cookbook `__ From 31495bd2d6df1ab6b84e480f49507424bc19d29d Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 29 Jan 2021 12:39:29 -0800 Subject: [PATCH 3/7] flyte-cli --- rsts/user/getting_started/create_first.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index e7b15e4282..a490eb4a71 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -22,9 +22,11 @@ Creating a Project In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project. -For example, to create a new project called ``myflyteproject``: use the project creation endpoint to create the new project :: +For example, to create a new project called ``myflyteproject``: in a virtual environment with flytekit installed use the +``flyte-cli`` to create the new project :: - curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "myflyteproject", "name": "myflyteproject"} }' + flyte-cli register-project -h localhost:30081 -i - myflyteproject --name "My Flyte Project" \ + --description "My very first project getting started on Flyte" Writing a Task From edf126cdd78644f7e6c9f17b04debcfa8bfab0a1 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 29 Jan 2021 14:09:53 -0800 Subject: [PATCH 4/7] comments --- rsts/user/getting_started/create_first.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index a490eb4a71..f930eb1233 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -72,8 +72,12 @@ From there, we can begin to write our first task. It should look something like Some of the new concepts demonstrated here are: -* Use the ``@task`` decorator to convert your typed python function to a Flyte task. -* A ``FlyteFile`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. +* Use the :py:func:`flytekit.task` decorator to convert your typed python function to a Flyte task. +* A :py:class:`flytekit.types.file.FlyteFile` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. + + +You can call this task ``edge_detection_canny(image_location=...)`` and iterate locally before adding it to part of a larger overall workflow. + Writing a Workflow ********************* @@ -90,7 +94,11 @@ This code block creates a workflow, with one task. The workflow itself has an in You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate locally before moving on to register it with Flyte. +.. note:: + + Every invocation of a Flyte workflow requires specifying keyword args. + Interacting with Flyte ************************ -For detailed and interactive example workflow 'recipes' check out the `Flytesnacks Cookbook `__ +If you're interested in learning more and want to try more complex examples, `Flytesnacks Cookbook `__ From a187f0e8993efd16eba98a740d742366631379d8 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 29 Jan 2021 14:15:28 -0800 Subject: [PATCH 5/7] docs --- rsts/user/getting_started/index.rst | 2 +- rsts/user/getting_started/more_examples.rst | 9 --------- 2 files changed, 1 insertion(+), 10 deletions(-) delete mode 100644 rsts/user/getting_started/more_examples.rst diff --git a/rsts/user/getting_started/index.rst b/rsts/user/getting_started/index.rst index 4d7a7a3bcd..0f75f6d18e 100644 --- a/rsts/user/getting_started/index.rst +++ b/rsts/user/getting_started/index.rst @@ -12,6 +12,6 @@ Getting Started :maxdepth: 1 :caption: Getting Started - examples create_first + examples Learn Flytekit by example diff --git a/rsts/user/getting_started/more_examples.rst b/rsts/user/getting_started/more_examples.rst deleted file mode 100644 index 84f23333f9..0000000000 --- a/rsts/user/getting_started/more_examples.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _getting-started-more-examples: - -######################################## -Want more example to Deep Dive -######################################## - -Getting started Guide and your first Workflow is just a tip of the iceberg. To understand and dive deeper and find a set of examples, we recommend going through -`FlyteSnacks `_. We are working on this as a interactive tutorial that you can run locally on your laptop. Make sure that your laptop is setup to run -Flyte. From 70bb71aa8aa3c0184e2e1aa31a29ba161a11d537 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Fri, 5 Feb 2021 14:55:18 -0800 Subject: [PATCH 6/7] more --- rsts/user/getting_started/create_first.rst | 40 +++++++++++----------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index f930eb1233..0677142c26 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -4,31 +4,18 @@ Writing Your First Workflow ######################################## +By the end of this getting started guide you'll be familiar with how easy it is to author a Flyte workflow, run and then deploy it locally. + The easiest way to author a Flyte Workflow is using the provided python SDK called "FlyteKit". -You can save some effort by cloning the ``flytesnacks`` repo, and re-initializing it as a new git repository :: +You can save some effort by cloning the ``flytekit-python-template`` repo, and re-initializing it as a new git repository :: - git clone git@github.com:lyft/flytesnacks.git myflyteproject + git clone git@github.com:lyft/flytekit-python-template.git myflyteproject cd myflyteproject rm -rf .git git init cd python - -Let's take a look at how easy it is to use Flyte by writing a task from scratch. - -Creating a Project -****************** - -In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project. - -For example, to create a new project called ``myflyteproject``: in a virtual environment with flytekit installed use the -``flyte-cli`` to create the new project :: - - flyte-cli register-project -h localhost:30081 -i - myflyteproject --name "My Flyte Project" \ - --description "My very first project getting started on Flyte" - - Writing a Task ***************** @@ -37,7 +24,9 @@ The most basic Flyte primitive is a "task". Flyte Tasks are units of work that c Start by creating a new file :: - touch cookbook/recipes/core/first.py + touch myapp/workflows/first.py + +And add the required imports we'll need for this example: .. code-block:: python @@ -73,10 +62,11 @@ From there, we can begin to write our first task. It should look something like Some of the new concepts demonstrated here are: * Use the :py:func:`flytekit.task` decorator to convert your typed python function to a Flyte task. -* A :py:class:`flytekit.types.file.FlyteFile` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. +* A :py:class:`flytekit.types.file.FlyteFile` is a Flytekit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. -You can call this task ``edge_detection_canny(image_location=...)`` and iterate locally before adding it to part of a larger overall workflow. +You can call this task ``edge_detection_canny(image_location="https://images.unsplash.com/photo-1512289984044-071903207f5e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=2250&q=80")`` and iterate locally before adding it to part of a larger overall workflow. + Writing a Workflow @@ -101,4 +91,14 @@ You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate local Interacting with Flyte ************************ +TODO: fill this section out. +1. Setup a sandbox deployment +2. Create a project +3. Register your workflows +4. Run your workflows + + +Expanded examples +***************** + If you're interested in learning more and want to try more complex examples, `Flytesnacks Cookbook `__ From b6ce900a174cb19d91fb3d9b7d63eacb53dcbbd7 Mon Sep 17 00:00:00 2001 From: Katrina Rogan Date: Mon, 8 Feb 2021 10:00:40 -0800 Subject: [PATCH 7/7] Update rsts/user/getting_started/create_first.rst Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> --- rsts/user/getting_started/create_first.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index 0677142c26..6ec8d1616d 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -4,7 +4,7 @@ Writing Your First Workflow ######################################## -By the end of this getting started guide you'll be familiar with how easy it is to author a Flyte workflow, run and then deploy it locally. +By the end of this getting started guide you'll be familiar with how easy it is to author a Flyte workflow, run and then deploy it to a sandbox cluster (which can be run locally or on any kubernetes cluster). The easiest way to author a Flyte Workflow is using the provided python SDK called "FlyteKit".