Skip to content

Commit

Permalink
Added intersphinx links to IDL docs (flyteorg#223)
Browse files Browse the repository at this point in the history
Signed-off-by: Prafulla Mahindrakar <[email protected]>
  • Loading branch information
pmahindrakar-oss authored May 20, 2021
1 parent 773a541 commit 42f818a
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 8 deletions.
8 changes: 4 additions & 4 deletions cookbook/core/extend_flyte/backend_plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ We will try to understand the components of a backend plugin using an example pl

Interface specification
------------------------
Usually Flyte extensions need information that is not covered by a :std:ref:`Flyte TaskTemplate <api_msg_flyteidl.core.tasktemplate>`. The TaskTemplate consists of a
the interface, task_type identifier, some metadata and other fields. An important field to note here is - :std:ref:`api_field_flyteidl.core.tasktemplate.custom`. The custom field is essentially an unstructured JSON.
This makes it possible to extend a task-template beyond the default supported targets -- :std:ref:`api_field_flyteidl.core.tasktemplate.container` (WIP, sql etc).
Usually Flyte extensions need information that is not covered by a :std:ref:`Flyte TaskTemplate <protos/docs/core/core:tasktemplate>`. The TaskTemplate consists of a
the interface, task_type identifier, some metadata and other fields. An important field to note here is - :std:ref:`custom <protos/docs/core/core:tasktemplate>`. The custom field is essentially an unstructured JSON.
This makes it possible to extend a task-template beyond the default supported targets -- :std:ref:`container <protos/docs/core/core:tasktemplate>` (WIP, sql etc).

The motivation of the Custom field, is to marshal a JSON structure that specifies information beyond what a regular TaskTemplate can capture. The actual structure of the JSON is known only to the implemented backend-plugin and the SDK components. The core Flyte platform, does not understand of look into the specifics of this structure.
It is highly recommended to use an interface definition lanugage like Protobuf, OpenAPISpec etc to declare specify the structure of the JSON. From here, on we refer to this as the ``Plugin Specification``.
Expand Down Expand Up @@ -72,4 +72,4 @@ Catch-all - Core Plugin
.. NOTE::

Coming soon 🛠


1 change: 1 addition & 0 deletions cookbook/docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -318,4 +318,5 @@ def __call__(self, filename):
"flyte": ("https://flyte.readthedocs.io/en/latest/", None),
# Uncomment for local development and change to your username
# "flytekit": ("/Users/ytong/go/src/github.com/lyft/flytekit/docs/build/html", None),
"flyteidl": ("https://docs.flyte.org/projects/flyteidl/en/latest", None),
}
8 changes: 4 additions & 4 deletions cookbook/docs/extend_flyte_backend_plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ We will try to understand the components of a backend plugin using an example pl

Interface specification
------------------------
Usually Flyte extensions need information that is not covered by a :std:ref:`Flyte TaskTemplate <api_msg_flyteidl.core.tasktemplate>`. The TaskTemplate consists of a
the interface, task_type identifier, some metadata and other fields. An important field to note here is - :std:ref:`api_field_flyteidl.core.tasktemplate.custom`. The custom field is essentially an unstructured JSON.
This makes it possible to extend a task-template beyond the default supported targets -- :std:ref:`api_field_flyteidl.core.tasktemplate.container` (WIP, sql etc).
Usually Flyte extensions need information that is not covered by a :std:ref:`Flyte TaskTemplate <protos/docs/core/core:tasktemplate>`. The TaskTemplate consists of a
the interface, task_type identifier, some metadata and other fields. An important field to note here is - :std:ref:`custom <protos/docs/core/core:tasktemplate>`. The custom field is essentially an unstructured JSON.
This makes it possible to extend a task-template beyond the default supported targets -- :std:ref:`container <protos/docs/core/core:tasktemplate>` (WIP, sql etc).

The motivation of the Custom field, is to marshal a JSON structure that specifies information beyond what a regular TaskTemplate can capture. The actual structure of the JSON is known only to the implemented backend-plugin and the SDK components. The core Flyte platform, does not understand of look into the specifics of this structure.
It is highly recommended to use an interface definition lanugage like Protobuf, OpenAPISpec etc to declare specify the structure of the JSON. From here, on we refer to this as the ``Plugin Specification``.

For Spark we decided to use Protobuf to specify the plugin as can be seen `here <https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/plugins/spark.proto>`__. Note it is not necessary to have the Plugin structure specified in FlyteIDL, we do it for simplicity, ease of maintenance alongwith the core platform and because of existing tooling to generate code for protobuf.
For Spark we decided to use Protobuf to specify the plugin as can be seen `here <protos/docs/plugins/plugins:flyteidl/plugins/spark.proto>`__. Note it is not necessary to have the Plugin structure specified in FlyteIDL, we do it for simplicity, ease of maintenance alongwith the core platform and because of existing tooling to generate code for protobuf.

Flytekit Plugin implementation
--------------------------------
Expand Down

0 comments on commit 42f818a

Please sign in to comment.