[Core feature] Offload parts of CRD to storage

[ckiosidis]

Motivation: Why do you think this is important?

The larger the execution the larger the flyteworkflow resource size.
At times we have observed flyteworkflow larger than 1.3MB, which is the upper size limit for crds in GKE.

We believe that updating large resources can also cause performance degradation issues.

Offloading some static parts of the CRD to storage (s3/gcs/etc) will help make the flyteworkflow lighter and more easily manageable.

Goal: What should the final outcome look like, ideally?

Reduce the size of flyteworkflow kubernetes resources for large Flyte Workflows.
Large flyte workflows can be workflows with a lot of nodes or with big subworkflows.

Flyteadmin would write the parts of the CRD to storage before creating the FlyteWorkflow resource on kubernetes.
Flytepropeller would read the CRD parts when handling the FlyteWorkflow resource.

Describe alternatives you've considered

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[ckiosidis]

Should this behaviour be configurable from flyteadmin via configuration?

[hamersaw]

Should this behaviour be configurable from flyteadmin via configuration?

I think so. To begin with, lets just add a top-level option which handles all workflows. If we find it would be helpful we could make this per-workflow as well.

[ckiosidis]

flyteorg/flytepropeller#477