Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update getting starting page with a minimal example & link to flytesnacks #682

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 43 additions & 64 deletions rsts/user/getting_started/create_first.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,19 @@ You can save some effort by cloning the ``flytesnacks`` repo, and re-initializin
git init
cd python
katrogan marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cd python
cd myflyteproject

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll just remove. we should already be in myflyteproject


now open the "Makefile" and change the first line to ``IMAGE_NAME=myflyteproject``

Let's also remove the existing python task so we can write one from scratch. ::

rm single_step/edges.py
Let's take a look at how easy it is to use Flyte by writing a task from scratch.

Creating a Project
******************

In Flyte, workflows are organized into namespaces called "Projects". When you register a workflow, it must be registered under a project.

Lets create a new project called ``myflyteproject``. Use the project creation endpoint to create the new project ::
For example, to create a new project called ``myflyteproject``: in a virtual environment with flytekit installed use the
``flyte-cli`` to create the new project ::

curl -X POST localhost:30081/api/v1/projects -d '{"project": {"id": "myflyteproject", "name": "myflyteproject"} }'
flyte-cli register-project -h localhost:30081 -i - myflyteproject --name "My Flyte Project" \
--description "My very first project getting started on Flyte"


Writing a Task
Expand All @@ -37,89 +36,69 @@ The most basic Flyte primitive is a "task". Flyte Tasks are units of work that c

Start by creating a new file ::

mkdir -p workflows
touch workflows/first.py

This directory has been marked in the `configuration file <https://github.com/lyft/flytesnacks/blob/764b82aca5701137ebc0eda4e818466e5acc9219/sandbox.config#L2>`_ as the location to look for workflows and tasks. Begin by importing some of the libraries that we'll need for this example.
touch cookbook/recipes/core/first.py


.. code-block:: python

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import urllib.request as _request
from urllib import request

import cv2
from flytekit.common import utils
from flytekit.sdk.tasks import python_task, outputs, inputs
from flytekit.sdk.types import Types
from flytekit.sdk.workflow import workflow_class, Output, Input
import flytekit
from flytekit import task, workflow
from flytekit.types.file import FlyteFile

From there, we can begin to write our first task. It should look something like this.
From there, we can begin to write our first task. It should look something like this:

.. code-block:: python

@inputs(image_location=Types.String)
@outputs(parsed_image=Types.Blob)
@python_task
def edge_detection_canny(wf_params, image_location, parsed_image):
with utils.AutoDeletingTempDir('test') as tmpdir:
plane_fname = '{}/plane.jpg'.format(tmpdir.name)
with _request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile:
data = d.read()
opfile.write(data)

img = cv2.imread(plane_fname, 0)
edges = cv2.Canny(img, 50, 200) # hysteresis thresholds

output_file = '{}/output.jpg'.format(tmpdir.name)
cv2.imwrite(output_file, edges)

parsed_image.set(output_file)
@task
def edge_detection_canny(image_location:str) -> FlyteFile:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like the image_location to be a Flytefile ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess we can do that as a follow up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we keeping this opencv example?

working_dir = flytekit.current_context().working_directory:
plane_fname = '{}/plane.jpg'.format(working_dir.name)
with request.urlopen(image_location) as d, open(plane_fname, 'wb') as opfile:
data = d.read()
opfile.write(data)

img = cv2.imread(plane_fname, 0)
edges = cv2.Canny(img, 50, 200) # hysteresis thresholds

Some of the new concepts demonstrated here are:
output_file = '{}/output.jpg'.format(working_dir.name)
cv2.imwrite(output_file, edges)

* ``wf_params``: The first argument to a python task is a Flyte SDK defined object that offers handlers like logging.
* Inputs and outputs are first defined in the decorator, and then passed into the argument of the function. Note that the names in the function signature need to match those in the decorator arguments.
* A ``Blob`` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image.
return FlyteFile["jpg"](path=output_file)
katrogan marked this conversation as resolved.
Show resolved Hide resolved

Writing a Workflow
*********************
Next you need to call that task from a workflow. In the same file, add these lines.

.. code-block:: python
Some of the new concepts demonstrated here are:

@workflow_class
class EdgeDetectorWf(object):
image_input = Input(Types.String, required=True, help="Image to run for")
run_edge_detection = edge_detection_canny(image_location=image_input)
edges = Output(run_edge_detection.outputs.parsed_image, sdk_type=Types.Blob)

This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image.
* Use the :py:func:`flytekit.task` decorator to convert your typed python function to a Flyte task.
* A :py:class:`flytekit.types.file.FlyteFile` is a Flyte Kit type that represents binary data. It is used to offload data to a storage location like S3. Here we use it to store an image.


Interacting with Flyte
************************
You can call this task ``edge_detection_canny(image_location=...)`` and iterate locally before adding it to part of a larger overall workflow.

Flyte fulfills tasks using docker images. You'll need to build a docker image from this code before it can run in Flyte. The repo has a make target to build the docker image for you ::

eval $(minikube -p minikube docker-env) # If using the flyte sandbox via minikube
make docker_build
Writing a Workflow
*********************
Next you need to call that task from a workflow. In the same file, add these lines.

If you have the flyte sandbox installed on your local machine, the image will be accessible to your Flyte system. If you're running a remote Flyte instance, you'll need to upload this image to a remote registry such as Dockerhub, Amazon ECR, or Google Container Registry, so that it can be used by the Flyte system.
.. code-block:: python

To upload to a remote registry (or even local registry), use ::
@workflow
class EdgeDetectorWf(image_input: str) -> FlyteFile:
edges = edge_detection_canny(image_location=image_input)
return edges

DOCKER_REGISTRY_USERNAME={username} DOCKER_REGISTRY_PASSWORD={pass} REGISTRY=docker.io make docker_build
This code block creates a workflow, with one task. The workflow itself has an input (the link to an image) that gets passed into the task, and an output, which is the processed image.

Replace the values above with your registry username, password, and registry endpoint.
You can call this workflow ``EdgeDetectorWf(image_input=...)`` and iterate locally before moving on to register it with Flyte.
katrogan marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, reason this is easy copy-pastable


You may need to change the ``IMAGE_NAME`` in the Makefile to reflect your namespace in the docker registry. (ex ``{{my docker username}}/myflyteproject``)
.. note::

With the image built, we just need to register the tasks and workflows. The process is the same as what we had done previously. ::
Every invocation of a Flyte workflow requires specifying keyword args.

docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' {{ your docker image }} pyflyte -p myflyteproject -d development -c sandbox.config register workflows
Interacting with Flyte
************************

After this, you should be able to visit the Flyte UI, and run the workflow as you did with ``flytesnacks`` previously.
If you're interested in learning more and want to try more complex examples, `Flytesnacks Cookbook <https://flytecookbook.readthedocs.io/en/latest/>`__
2 changes: 1 addition & 1 deletion rsts/user/getting_started/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ Getting Started
:maxdepth: 1
:caption: Getting Started

examples
create_first
examples
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you asked to change the order?

Learn Flytekit by example <https://flytecookbook.readthedocs.io/en/latest/>
9 changes: 0 additions & 9 deletions rsts/user/getting_started/more_examples.rst

This file was deleted.