-
Notifications
You must be signed in to change notification settings - Fork 300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Customized container tasks and Shim tasks/executors #449
Conversation
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]> rename sqlalchemy entrypoint flyte task Signed-off-by: wild-endeavor <[email protected]> init Signed-off-by: wild-endeavor <[email protected]> bump Signed-off-by: wild-endeavor <[email protected]>
ee4b12c
to
760da2b
Compare
return python_types | ||
|
||
@classmethod | ||
def guess_python_type(cls, flyte_type: LiteralType) -> type: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we have to add it to every transformer now - or is there a default behavior
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
default behavior is to raise a ValueError
which gets caught and ignored.
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #449 +/- ##
==========================================
+ Coverage 84.50% 84.56% +0.05%
==========================================
Files 360 362 +2
Lines 26896 27009 +113
Branches 2214 2227 +13
==========================================
+ Hits 22729 22839 +110
- Misses 3567 3568 +1
- Partials 600 602 +2
Continue to review full report at Codecov.
|
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
return obj | ||
|
||
|
||
class TaskTemplateResolver(TrackedInstance, TaskResolverMixin): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why this rant
a task resolver was supposed to
* Be able to restore the same task when ``load_task`` is called as the object that ``loader_args`` was called on.
That is, even though at run time it's in a container on a cluster and is obviously a different Python process,
the Python object in memory should look the same.
* Offer a one-to-one mapping between the list of strings returned by the ``loader_args`` function, an the task,
at least within the container.```
Do we have this contract explicitly spelled out somewhere - if not this seems like a not so useful comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about - special resolver that resolves the ThirdPartyTask at runtime, using only the tasktemplate. Thus for any task, derived from x, this resolver is used and such tasks should be fully rehydrated from the task template alone
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i didn't think it was a rant. i'll add this comment and change the wording a bit.
|
||
|
||
@dataclass | ||
class SQLite3Config(object): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about use asciiflow to explain whats happening in a SQLite3Task or just in general for a ThirdPartContainer task?
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Wonderful. Thank you for all the work @wild-endeavor. quick question - would it be better to use ExecutableShimTask even in the local flow? |
TL;DR
This PR introduces the notion of task container plugins (aka flytekit only plugins). That is, you can now write and publish a Flyte Tasks for others to use, that when run, run a Docker container that you (the author of the plugin) specifies, rather than the user container. That is, typically, when a user writes an
@task
task, at serialization time, the container image that gets associated with that task is the main workflow/task image, the image made by the person writing the@task
decorator. This new construct allows the task author to tailor and control the execution container. The plugin author only has access to theTaskTemplate
of course, since the user code won't exist in the author's image.This relies on previous work done to expose the TaskTemplate to the execution end.
Type
Are all requirements met?
Complete description
Usage
As a plugin-task author
Follow the sqlite3 external container example as a template. Basically you need to
PythonCustomizedContainerTask
get_custom
function - this object will be serialized into theTaskTemplate
that the executor ultimately receives so put all the information you need to run that specific instance of the task in here. You can also override any other functions in the base class you need to.ShimTaskExecutor
and implementing theexecute_from_model
function.executor_type
argument to your executor subclass. An instance of this class will be created at execution time.As a user
Simply import the task from the Python library that provides it, and use as normal. The task will be registered and run as normal, just picking up the task author's container instead of yours at platform run-time. It is possible that the plugin author did not provide execution behavior in the importable Python task and it is only available in the container. In that case you'll need to mock it out withe
flytekit.patch
in order to run it locally.High Level Changes
ExecutableTemplateShimTask
.ShimTaskExecutor
.PythonThirdPartyContainerTask
which first uses aTaskTemplateResolver
implementing theTaskResolverMixin
interface to load the task, and then theShimTaskExecutor
object to run it.SQLite3Task
has been copied into a new folder underextras
. I was going to add it somewhere else, but I actually think having both versions around so that people can compare between them is better.TypeEngine
extended to add a guessing function to go backwards, from Flyte Literal types to Python types.Additional Changes
TaskResolverMixin
has been moved from the autocontainer file tobase_task.py
file and function signatures updated to reflect lower level classes.get_command
function inPythonFunctionTask
andPythonInstanceTask
were the same so this has been abstracted into the base class.entrypoint.py
cleaned up a bit, setup step was extracted out.Tracking Issue
NA