[Flytekit] Control Plane objects for native typed API

[wild-endeavor]

Revamp control plane objects

TL;DR

Introduce new control plane objects to go along with the new api. This is the last major component of the new api. (We won't gate removing the "beta" version on this issue though.)

Background

As part of the work on supporting the new flytekit API, the API that relies on native Python PEP 484 type hints available as of flytekit v0.16 and later, we've introduced a new type engine and new Python classes to represent tasks, workflows, launch plans, nodes, etc. In the older api, these ideas also existed - they were usually titled with the word Runnable in the class name and were sub-classes of the control plane objects (e.g. SdkRunnableTask inherits from SdkTask). The new api does not follow that inheritance idea - while building out the new code we thought that the writing of a task (or workflow) was a different enough construct than working with an existing task (or workflow) that the objects don't need to inherit from one another. A better pattern maybe is to tie both to the base model (for the new tasks, this tie is through the translator).

The feature-set offered by the extant Sdk classes is still required however. This ticket encompasses that entire effort.

Implementation

The current thinking is to replicate all the objects to new classes. This is not ideal, but the issue is we have no way of knowing what existing user code across all Flyte deployments contains. The ability of Python to be extended and modified also means the current classes may be used in ways we can't predict or take into account. To avoid breaking an unknown number of users, it's safer to just make new classes.

Items

Main things to do

• Create new objects named Flyte... to drive home the fact that these are not necessarily python/container/etc tasks. They just represent an entity on the Flyte backend.
• Use the new type engine. The current Sdk... tasks use the old type engine.
• Also, new classes should not use the old metaclasses.
  will add to this list as we discover them in implementation.

Features

What are the user interactions these objects should allow a user to do?

• Be able to fetch a workflow execution and inspect inputs/outputs at every node, and at the workflow layer.
• Be able to monitor the status of a workflow execution.
• Fetch a task/workflow and be able to understand its interface. And for tasks the task type.
• Compare 2 workflow executions (outputs)
• List recent workflow executions for a given project/domain/workflow name.
• Launch an execution of a task
• Launch an execution of a workflow
• Register a workflow from a file

Questions

• Should we support the usage of fetched tasks in new @workflows? This might not be a trivial thing since the new type engine relies on having both the Flyte type and the native Python type. With fetched tasks, we'll only have one side and will have to divine the other. There might not always be a one to one mapping. (We can start out by not allowing this, and add it in the future as well.)

[cosmicBboy]

hey @wild-endeavor I broke out this task into more granular ones:

• [Flytekit] Use new TypeEngine to convert flyte literals to python types in new Control Plane classes #974
• [Flytekit] Support subworkflows in new control plane classes #975
• [Flytekit] Support single task executions in new control plane classes #976
• [Flytekit] Centralized Control Plane Class to handle fetching, launching of flyte entities #973

Feel free to add more details/edits to those as needed

[wild-endeavor]

will make a new ticket for the remaining item