Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]Raw output path should be user configurable and sharding should not be necessary #486

Closed
1 of 13 tasks
kumare3 opened this issue Aug 27, 2020 · 0 comments
Closed
1 of 13 tasks

[Feature]Raw output path should be user configurable and sharding should not be necessary #486

kumare3 opened this issue Aug 27, 2020 · 0 comments
Assignees
@kumare3
Labels
enhancement New feature or request
Milestone
0.11.0

Comments

@kumare3
Copy link
Contributor

kumare3 commented Aug 27, 2020

Motivation: Why do you think this is important?
Currently Raw output path generated by FlytePropeller (or flytekit) are always sharded using a 2 char prefix. This is required only for presharded S3 buckets and not useful globally. This also makes the path obfuscated and where throughput is not a constraint that might be desirable.

Goal: What should the final outcome look like, ideally?
User will be able to specify the path strategy as - "human-readable, short, sharded" in the launchplan declaration and that should happen automatically

Describe alternatives you've considered
Not possible today

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline
NA (TO Be Done)

Additional context
NA

Is this a blocker for you to adopt Flyte
NO
@kumare3 kumare3 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Aug 27, 2020
@kumare3 kumare3 mentioned this issue Aug 27, 2020
Merged
7 tasks
@kumare3 kumare3 self-assigned this Jan 29, 2021
@kumare3 kumare3 removed the untriaged This issues has not yet been looked at by the Maintainers label Jan 29, 2021
@kumare3 kumare3 added this to the 0.11.0 milestone Jan 29, 2021
@kumare3 kumare3 closed this as completed Jan 29, 2021
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@andrewwdye @flyte-bot
@EngHabu @hamersaw
Allow checkpoint resume when recovering a workflow (flyteorg#486)
a9b831b 
* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Fix build break

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Save/restore CheckpointUri from NodeExecution

Signed-off-by: Andrew Dye <[email protected]>

* Lints, generate

Signed-off-by: Andrew Dye <[email protected]>

* Fix log line

Signed-off-by: Andrew Dye <[email protected]>

Signed-off-by: Flyte-Bot <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Andrew Dye <[email protected]>
Co-authored-by: flyte-bot <[email protected]>
Co-authored-by: Haytham Abuelfutuh <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@samhita-alla
Add ShellTask Example (flyteorg#486)
6cb5ad6
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@apTalya @anrusina
chore: update Readme with python note + package version update (flyte…
821299e 
…org#486)

Signed-off-by: Nastya Rusina <[email protected]>

Co-authored-by: Nastya Rusina <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@andrewwdye @flyte-bot
@EngHabu @hamersaw
Allow checkpoint resume when recovering a workflow (flyteorg#486)
7d6d09d 
* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Fix build break

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update flyteidl version

Signed-off-by: Flyte-Bot <[email protected]>

* Save/restore CheckpointUri from NodeExecution

Signed-off-by: Andrew Dye <[email protected]>

* Lints, generate

Signed-off-by: Andrew Dye <[email protected]>

* Fix log line

Signed-off-by: Andrew Dye <[email protected]>

Signed-off-by: Flyte-Bot <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Andrew Dye <[email protected]>
Co-authored-by: flyte-bot <[email protected]>
Co-authored-by: Haytham Abuelfutuh <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kumare3 kumare3

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.11.0
Development

No branches or pull requests
1 participant
@kumare3