diff --git a/rsts/user/getting_started/create_first.rst b/rsts/user/getting_started/create_first.rst index 2ae9b8129f..6ec8d1616d 100644 --- a/rsts/user/getting_started/create_first.rst +++ b/rsts/user/getting_started/create_first.rst @@ -4,32 +4,18 @@ Writing Your First Workflow ######################################## +By the end of this getting started guide you'll be familiar with how easy it is to author a Flyte workflow, run and then deploy it to a sandbox cluster (which can be run locally or on any kubernetes cluster). + The easiest way to author a Flyte Workflow is using the provided python SDK called "FlyteKit". -You can save some effort by cloning the ``flytesnacks`` repo, and re-initializing it as a new git repository :: +You can save some effort by cloning the ``flytekit-python-template`` repo, and re-initializing it as a new git repository :: - git clone git@github.com:lyft/flytesnacks.git myflyteproject + git clone git@github.com:lyft/flytekit-python-template.git myflyteproject cd myflyteproject rm -rf .git git init cd python -now open the "Makefile" and change the first line to ``IMAGE_NAME=myflyteproject`` - -Let's also remove the existing python task so we can write one from scratch. :: - - rm single_step/edges.py - -Creating a Project -****************** - -In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project. - -Lets create a new project called ``myflyteproject``. Use the project creation endpoint to create the new project :: - - curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "myflyteproject", "name": "myflyteproject"} }' - - Writing a Task ***************** @@ -37,53 +23,51 @@ The most basic Flyte primitive is a "task". Flyte Tasks are units of work that c Start by creating a new file :: - mkdir -p workflows - touch workflows/first.py -This directory has been marked in the `configuration file `_ as the location to look for workflows and tasks. Begin by importing some of the libraries that we'll need for this example. + touch myapp/workflows/first.py + +And add the required imports we'll need for this example: + .. code-block:: python - from __future__ import absolute_import - from __future__ import division - from __future__ import print_function - - import urllib.request as _request + from urllib import request import cv2 - from flytekit.common import utils - from flytekit.sdk.tasks import python_task, outputs, inputs - from flytekit.sdk.types import Types - from flytekit.sdk.workflow import workflow_class, Output, Input + import flytekit + from flytekit import task, workflow + from flytekit.types.file import FlyteFile -From there, we can begin to write our first task. It should look something like this. +From there, we can begin to write our first task. It should look something like this: .. code-block:: python - @inputs(image_location=Types.String) - @outputs(parsed_image=Types.Blob) - @python_task - def edge_detection_canny(wf_params, image_location, parsed_image): - with utils.AutoDeletingTempDir('test') as tmpdir: - plane_fname = '{}/plane.jpg'.format(tmpdir.name) - with _request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile: - data = d.read() - opfile.write(data) - - img = cv2.imread(plane_fname, 0) - edges = cv2.Canny(img, 50, 200) # hysteresis thresholds - - output_file = '{}/output.jpg'.format(tmpdir.name) - cv2.imwrite(output_file, edges) - - parsed_image.set(output_file) + @task + def edge_detection_canny(image_location:str) -> FlyteFile: + working_dir = flytekit.current_context().working_directory: + plane_fname = '{}/plane.jpg'.format(working_dir.name) + with request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile: + data = d.read() + opfile.write(data) + + img = cv2.imread(plane_fname, 0) + edges = cv2.Canny(img, 50, 200) # hysteresis thresholds + + output_file = '{}/output.jpg'.format(working_dir.name) + cv2.imwrite(output_file, edges) + + return FlyteFile["jpg"](path=output_file) Some of the new concepts demonstrated here are: -* ``wf_params``: The first argument to a python task is a Flyte SDK defined object that offers handlers like logging. -* Inputs and outputs are first defined in the decorator, and then passed into the argument of the function. Note that the names in the function signature need to match those in the decorator arguments. -* A ``Blob`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. +* Use the :py:func:`flytekit.task` decorator to convert your typed python function to a Flyte task. +* A :py:class:`flytekit.types.file.FlyteFile` is a Flytekit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image. + + +You can call this task ``edge_detection_canny(image_location="https://images.unsplash.com/photo-1512289984044-071903207f5e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=2250&q=80")`` and iterate locally before adding it to part of a larger overall workflow. + + Writing a Workflow ********************* @@ -91,35 +75,30 @@ Next you need to call that task from a workflow. In the same file, add these li .. code-block:: python - @workflow_class - class EdgeDetectorWf(object): - image_input = Input(Types.String, required=True, help="Image to run for") - run_edge_detection = edge_detection_canny(image_location=image_input) - edges = Output(run_edge_detection.outputs.parsed_image, sdk_type=Types.Blob) - -This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image. - + @workflow + class EdgeDetectorWf(image_input: str) -> FlyteFile: + edges = edge_detection_canny(image_location=image_input) + return edges -Interacting with Flyte -************************ - -Flyte fulfills tasks using docker images. You'll need to build a docker image from this code before it can run in Flyte. The repo has a make target to build the docker image for you :: - - eval $(minikube -p minikube docker-env) # If using the flyte sandbox via minikube - make docker_build +This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image. -If you have the flyte sandbox installed on your local machine, the image will be accessible to your Flyte system. If you're running a remote Flyte instance, you'll need to upload this image to a remote registry such as Dockerhub, Amazon ECR, or Google Container Registry, so that it can be used by the Flyte system. +You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate locally before moving on to register it with Flyte. -To upload to a remote registry (or even local registry), use :: +.. note:: - DOCKER_REGISTRY_USERNAME={username} DOCKER_REGISTRY_PASSWORD={pass} REGISTRY=docker.io make docker_build + Every invocation of a Flyte workflow requires specifying keyword args. -Replace the values above with your registry username, password, and registry endpoint. +Interacting with Flyte +************************ -You may need to change the ``IMAGE_NAME`` in the Makefile to reflect your namespace in the docker registry. (ex ``{{my docker username}}/myflyteproject``) +TODO: fill this section out. +1. Setup a sandbox deployment +2. Create a project +3. Register your workflows +4. Run your workflows -With the image built, we just need to register the tasks and workflows. The process is the same as what we had done previously. :: - docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' {{ your docker image }} pyflyte -p myflyteproject -d development -c sandbox.config register workflows +Expanded examples +***************** -After this, you should be able to visit the Flyte UI, and run the workflow as you did with ``flytesnacks`` previously. +If you're interested in learning more and want to try more complex examples, `Flytesnacks Cookbook `__ diff --git a/rsts/user/getting_started/index.rst b/rsts/user/getting_started/index.rst index 4d7a7a3bcd..0f75f6d18e 100644 --- a/rsts/user/getting_started/index.rst +++ b/rsts/user/getting_started/index.rst @@ -12,6 +12,6 @@ Getting Started :maxdepth: 1 :caption: Getting Started - examples create_first + examples Learn Flytekit by example diff --git a/rsts/user/getting_started/more_examples.rst b/rsts/user/getting_started/more_examples.rst deleted file mode 100644 index 84f23333f9..0000000000 --- a/rsts/user/getting_started/more_examples.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _getting-started-more-examples: - -######################################## -Want more example to Deep Dive -######################################## - -Getting started Guide and your first Workflow is just a tip of the iceberg. To understand and dive deeper and find a set of examples, we recommend going through -`FlyteSnacks `_. We are working on this as a interactive tutorial that you can run locally on your laptop. Make sure that your laptop is setup to run -Flyte.