Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core Feature] FlyteAdmin/Propeller should allow offloaded workflow specs #722

Closed
kumare3 opened this issue Feb 11, 2021 · 1 comment
Closed
Labels
enhancement New feature or request propeller Issues related to flyte propeller

Comments

@kumare3
Copy link
Contributor

kumare3 commented Feb 11, 2021

Motivation: Why do you think this is important?
There are cases in which the spec increases to a very large size - workflowtemplate or tasktempaltes with queries are very large. Also having the spec in etcD does not really offer any benefit.

Goal: What should the final outcome look like, ideally?
When a user executes a workflow, instead of writing the entire workflow to the CRD, we could write the spec to the Blob store and just pass a reference. This would have following benefits

  1. Extremely large workflow graphs (today Flyte supports workflow graphs in order of 5k, but this is subject to size of a task). This could increase graph to be arbitrarily large.
  2. Faster sync loops with etcD. This will improve transition performance

Describe alternatives you've considered
Spec compression. This is very complicated and debugging will be harder.

[Optional] Propose: Link/Inline OR Additional context
Instead of the inline spec - we can just have a reference to the spec in blob store - https://github.com/flyteorg/flytepropeller/blob/master/pkg/apis/flyteworkflow/v1alpha1/workflow.go#L29

@kumare3 kumare3 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers propeller Issues related to flyte propeller labels Feb 11, 2021
@yindia yindia self-assigned this Feb 15, 2021
@kumare3 kumare3 removed the untriaged This issues has not yet been looked at by the Maintainers label Mar 1, 2021
@kumare3 kumare3 added this to the 0.13.0 milestone Mar 1, 2021
@kumare3 kumare3 modified the milestones: 0.13.0, 0.15.0 May 5, 2021
@kumare3 kumare3 modified the milestones: 0.15.0, 0.16.0 Jun 7, 2021
@EngHabu EngHabu modified the milestones: 0.16.0, 0.17.0 Aug 2, 2021
@EngHabu EngHabu removed this from the 0.17.0 milestone Aug 31, 2021
palchicz pushed a commit to palchicz/flyte that referenced this issue Dec 23, 2021
* schema in dataclass

Signed-off-by: Kevin Su <[email protected]>

* Added tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed lint

Signed-off-by: Kevin Su <[email protected]>

* Updated tests

Signed-off-by: Kevin Su <[email protected]>

* Updated tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed lint

Signed-off-by: Kevin Su <[email protected]>

* updated

Signed-off-by: Kevin Su <[email protected]>

* updated

Signed-off-by: Kevin Su <[email protected]>

* updated

Signed-off-by: Kevin Su <[email protected]>
@hamersaw
Copy link
Contributor

Completed in #2705

eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
* updat image

Signed-off-by: Yee Hing Tong <[email protected]>

* update sha again

Signed-off-by: Yee Hing Tong <[email protected]>

* delete flytekitvenv

Signed-off-by: Yee Hing Tong <[email protected]>

* fixes

Signed-off-by: Yee Hing Tong <[email protected]>

* delete bad file

Signed-off-by: Yee Hing Tong <[email protected]>

* revert 4.02

Signed-off-by: Yee Hing Tong <[email protected]>

* reverttagdeletion

Signed-off-by: Yee Hing Tong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request propeller Issues related to flyte propeller
Projects
None yet
Development

No branches or pull requests

4 participants