Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] [FlyteKit] Intratask-checkpointing #709

Closed
1 of 13 tasks
EngHabu opened this issue Feb 8, 2021 · 1 comment
Closed
1 of 13 tasks

[Feature] [FlyteKit] Intratask-checkpointing #709

EngHabu opened this issue Feb 8, 2021 · 1 comment
Labels
enhancement New feature or request flytekit FlyteKit Python related issue untriaged This issues has not yet been looked at by the Maintainers

Comments

@EngHabu
Copy link
Contributor

EngHabu commented Feb 8, 2021

Motivation: Why do you think this is important?
Flyte natively supports at-task-boundry checkpointing. Intratask checkpointing aims at providing a way for users to checkpoint at various stages during the task execution.

Goal: What should the final outcome look like, ideally?
The goal is to support a way for task authors to signal what data is ready to be checkpointed and for Flyte to automatically make the data available on task retry.

This is particularly useful for:

  • Crash recovery
  • Losing a Spot-instance (Lower cost machines based on price betting) and recovering from such failures.

Spec: TBD

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other
@EngHabu EngHabu added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers plugins Plugins related labels (backend or frontend) labels Feb 8, 2021
@kumare3 kumare3 added flytekit FlyteKit Python related issue and removed plugins Plugins related labels (backend or frontend) labels Feb 9, 2021
@EngHabu
Copy link
Contributor Author

EngHabu commented Feb 9, 2021

Dupe: #634

@EngHabu EngHabu closed this as completed Feb 9, 2021
@sbrunk sbrunk mentioned this issue Mar 14, 2022
Closed
2 tasks
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@SmritiSatyanV
Render a specific section in knn_classifier.py (flyteorg#709)
38c8590 
* Updated knn_classifier

Signed-off-by: SmritiSatyanV <[email protected]>

* change

Signed-off-by: SmritiSatyanV <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request flytekit FlyteKit Python related issue untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EngHabu @kumare3