Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Keep processing the same workflow as long as there are changes #676

Closed
1 of 13 tasks
EngHabu opened this issue Jan 23, 2021 · 1 comment
Closed
1 of 13 tasks
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@EngHabu
Copy link
Contributor

EngHabu commented Jan 23, 2021

Motivation: Why do you think this is important?
It can speed up progressing workflows drastically.

Goal: What should the final outcome look like, ideally?
Propeller's algorithm today does the following:

  • Sync WFs from informer
  • Iterate on WFs, traverse each and attempt to make progress
  • Store
  • Wait for next sync period.

With the change, it would:
1- Sync WFs from informer
2- Iterate on WFs
3- Traverse each and attempt to make progress
4- Store
5- If a change is detected (https://github.com/lyft/flytepropeller/blob/master/pkg/controller/handler.go#L212), go back to 3, if not, go to 6
6- Wait for next sync period.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other
@EngHabu EngHabu added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Jan 23, 2021
@kumare3 kumare3 self-assigned this Jan 29, 2021
@kumare3 kumare3 removed the untriaged This issues has not yet been looked at by the Maintainers label Jan 29, 2021
@kumare3 kumare3 added this to the 0.11.0 milestone Jan 29, 2021
@kumare3
Copy link
Contributor

kumare3 commented Feb 16, 2021

This has been merged and tested at scale. WE can observe and order of magnitude performance imrpovements

@kumare3 kumare3 closed this as completed Feb 16, 2021
eapolinario added a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants