Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add customizable resource contributor docs #514

Merged
merged 1 commit into from
Sep 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions rsts/contributor/components/customizable_resources.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
.. _components-customizable-resources:

#############################
Adding customizable resources
#############################

For background on customizable resources, see :ref:`managing_customizable_resources`. As a quick refresher, custom resources allow you to manage configurations for specific combinations of user projects, domains and workflows that override default values. Examples of such resources include execution clusters, task resource defaults, and `more <https://github.com/lyft/flyteidl/blob/master/protos/flyteidl/admin/matchable_resource.proto>`__.


Example
-------

Let's say you want to inject a default priority annotation for your workflows. Perhaps you start off with a model where everything has a default priority but soon you realize it makes sense that workflows in your production domain should take higher priority than those in your development domain.

Now one of your user teams comes to you and wants to ensure that their critical workflows have even higher priority than other production workflows. Here's how you could go about building a customizable priority designation.

Flyte IDL
^^^^^^^^^
You'll want to introduce a new `matchable attribute <https://github.com/lyft/flyteidl/blob/master/protos/flyteidl/admin/matchable_resource.proto>`__, including a unique enum value and proto message definition. For example

::

enum MatchableResource {
...
WORKFLOW_PRIORITY = 10;
}

message WorkflowPriorityAttribute {
int priority = 1;
}

message MatchingAttributes {
oneof target {
...
WorkflowPriorityAttribute WorkflowPriority = 11;
}
}


See the changes in this `file <https://github.com/lyft/flyteidl/commit/b1767697705621a3fddcb332617a5304beba5bec#diff-d3c1945436aba8f7a76755d75d18e671>`__ for an example of what is required.


Flyte admin
^^^^^^^^^^^

Once your idl changes are released, update the logic of flyteadmin to `fetch <https://github.com/lyft/flyteadmin/commit/60b4c876ea105d4c79e3cad7d56fde6b9c208bcd#diff-510e72225172f518850fe582149ff320R122-R128>`__ your new matchable priority resource and use it when creating executions or wherever makes sense for your use case.

For example:

::


resource, err := s.resourceManager.GetResource(ctx, managerInterfaces.ResourceRequest{
Domain: domain,
Project: project, // optional
Workflow: workflow, // optional, must include project when specifying workflow
LaunchPlan: launchPlan, // optional, must include project + workflow when specifying launch plan
ResourceType: admin.MatchableResource_WORKFLOW_PRIORITY,
})

if err != nil {
return err
}

if resource != nil && resource.Attributes != nil && resource.Attributes.GetWorkflowPriority() != nil {
priorityValue := resource.Attributes.GetWorkflowPriority().GetPriority()
// do something with the priority here
}


Flytekit
^^^^^^^^
For convenience, add a flyte-cli wrapper for updating your new attribute. See `this PR <https://github.com/lyft/flytekit/pull/174>`__ for the full required set of changes.

That's it! You now have a new matchable attribute you can configure as the needs of your users evolve.
1 change: 1 addition & 0 deletions rsts/contributor/components/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ These sections provide implementation details about specific flyte components. T
admin_service
catalog
console
customizable_resources