Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update getting starting page with a minimal example & link to flytesnacks #682

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 53 additions & 74 deletions rsts/user/getting_started/create_first.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,122 +4,101 @@
Writing Your First Workflow
########################################

By the end of this getting started guide you'll be familiar with how easy it is to author a Flyte workflow, run and then deploy it to a sandbox cluster (which can be run locally or on any kubernetes cluster).

The easiest way to author a Flyte Workflow is using the provided python SDK called "FlyteKit".

You can save some effort by cloning the ``flytesnacks`` repo, and re-initializing it as a new git repository ::
You can save some effort by cloning the ``flytekit-python-template`` repo, and re-initializing it as a new git repository ::

git clone [email protected]:lyft/flytesnacks.git myflyteproject
git clone [email protected]:lyft/flytekit-python-template.git myflyteproject
cd myflyteproject
rm -rf .git
git init
cd python
katrogan marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cd python
cd myflyteproject

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll just remove. we should already be in myflyteproject


now open the "Makefile" and change the first line to ``IMAGE_NAME=myflyteproject``

Let's also remove the existing python task so we can write one from scratch. ::

rm single_step/edges.py

Creating a Project
******************

In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project.

Lets create a new project called ``myflyteproject``. Use the project creation endpoint to create the new project ::

curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "myflyteproject", "name": "myflyteproject"} }'


Writing a Task
*****************

The most basic Flyte primitive is a "task". Flyte Tasks are units of work that can be composed in a workflow. The simplest way to write a Flyte task is using the Flyte Python SDK - flytekit.

Start by creating a new file ::

mkdir -p workflows
touch workflows/first.py

This directory has been marked in the `configuration file <https://github.com/lyft/flytesnacks/blob/764b82aca5701137ebc0eda4e818466e5acc9219/sandbox.config#L2>`_ as the location to look for workflows and tasks. Begin by importing some of the libraries that we'll need for this example.
touch myapp/workflows/first.py

And add the required imports we'll need for this example:


.. code-block:: python

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import urllib.request as _request
from urllib import request

import cv2
from flytekit.common import utils
from flytekit.sdk.tasks import python_task, outputs, inputs
from flytekit.sdk.types import Types
from flytekit.sdk.workflow import workflow_class, Output, Input
import flytekit
from flytekit import task, workflow
from flytekit.types.file import FlyteFile

From there, we can begin to write our first task. It should look something like this.
From there, we can begin to write our first task. It should look something like this:

.. code-block:: python

@inputs(image_location=Types.String)
@outputs(parsed_image=Types.Blob)
@python_task
def edge_detection_canny(wf_params, image_location, parsed_image):
with utils.AutoDeletingTempDir('test') as tmpdir:
plane_fname = '{}/plane.jpg'.format(tmpdir.name)
with _request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile:
data = d.read()
opfile.write(data)

img = cv2.imread(plane_fname, 0)
edges = cv2.Canny(img, 50, 200) # hysteresis thresholds

output_file = '{}/output.jpg'.format(tmpdir.name)
cv2.imwrite(output_file, edges)

parsed_image.set(output_file)
@task
def edge_detection_canny(image_location:str) -> FlyteFile:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like the image_location to be a Flytefile ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess we can do that as a follow up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we keeping this opencv example?

working_dir = flytekit.current_context().working_directory:
plane_fname = '{}/plane.jpg'.format(working_dir.name)
with request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile:
data = d.read()
opfile.write(data)

img = cv2.imread(plane_fname, 0)
edges = cv2.Canny(img, 50, 200) # hysteresis thresholds

output_file = '{}/output.jpg'.format(working_dir.name)
cv2.imwrite(output_file, edges)

return FlyteFile["jpg"](path=output_file)
katrogan marked this conversation as resolved.
Show resolved Hide resolved


Some of the new concepts demonstrated here are:

* ``wf_params``: The first argument to a python task is a Flyte SDK defined object that offers handlers like logging.
* Inputs and outputs are first defined in the decorator, and then passed into the argument of the function. Note that the names in the function signature need to match those in the decorator arguments.
* A ``Blob`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image.
* Use the :py:func:`flytekit.task` decorator to convert your typed python function to a Flyte task.
* A :py:class:`flytekit.types.file.FlyteFile` is a Flytekit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image.


You can call this task ``edge_detection_canny(image_location="https://images.unsplash.com/photo-1512289984044-071903207f5e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=2250&q=80")`` and iterate locally before adding it to part of a larger overall workflow.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to make this a

.. code-block::




Writing a Workflow
*********************
Next you need to call that task from a workflow. In the same file, add these lines.

.. code-block:: python

@workflow_class
class EdgeDetectorWf(object):
image_input = Input(Types.String, required=True, help="Image to run for")
run_edge_detection = edge_detection_canny(image_location=image_input)
edges = Output(run_edge_detection.outputs.parsed_image, sdk_type=Types.Blob)

This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image.

@workflow
class EdgeDetectorWf(image_input: str) -> FlyteFile:
edges = edge_detection_canny(image_location=image_input)
return edges

Interacting with Flyte
************************

Flyte fulfills tasks using docker images. You'll need to build a docker image from this code before it can run in Flyte. The repo has a make target to build the docker image for you ::

eval $(minikube -p minikube docker-env) # If using the flyte sandbox via minikube
make docker_build
This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image.

If you have the flyte sandbox installed on your local machine, the image will be accessible to your Flyte system. If you're running a remote Flyte instance, you'll need to upload this image to a remote registry such as Dockerhub, Amazon ECR, or Google Container Registry, so that it can be used by the Flyte system.
You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate locally before moving on to register it with Flyte.
katrogan marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, reason this is easy copy-pastable


To upload to a remote registry (or even local registry), use ::
.. note::

DOCKER_REGISTRY_USERNAME={username} DOCKER_REGISTRY_PASSWORD={pass} REGISTRY=docker.io make docker_build
Every invocation of a Flyte workflow requires specifying keyword args.

Replace the values above with your registry username, password, and registry endpoint.
Interacting with Flyte
************************

You may need to change the ``IMAGE_NAME`` in the Makefile to reflect your namespace in the docker registry. (ex ``{{my docker username}}/myflyteproject``)
TODO: fill this section out.
1. Setup a sandbox deployment
2. Create a project
3. Register your workflows
4. Run your workflows

With the image built, we just need to register the tasks and workflows. The process is the same as what we had done previously. ::

docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' {{ your docker image }} pyflyte -p myflyteproject -d development -c sandbox.config register workflows
Expanded examples
*****************

After this, you should be able to visit the Flyte UI, and run the workflow as you did with ``flytesnacks`` previously.
If you're interested in learning more and want to try more complex examples, `Flytesnacks Cookbook <https://flytecookbook.readthedocs.io/en/latest/>`__
2 changes: 1 addition & 1 deletion rsts/user/getting_started/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ Getting Started
:maxdepth: 1
:caption: Getting Started

examples
create_first
examples
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you asked to change the order?

Learn Flytekit by example <https://flytecookbook.readthedocs.io/en/latest/>
9 changes: 0 additions & 9 deletions rsts/user/getting_started/more_examples.rst

This file was deleted.