Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Get*ExecutionData should optionally return data inline #419

Closed
4 of 13 tasks
katrogan opened this issue Jul 20, 2020 · 1 comment
Closed
4 of 13 tasks

[Feature] Get*ExecutionData should optionally return data inline #419

katrogan opened this issue Jul 20, 2020 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@katrogan
Copy link
Contributor

Motivation: Why do you think this is important?
The current model of fetching signed URL references to execution data [inputs + outputs] works fine in a hosted AWS environment but doesn't actually successfully return a usable link in local/sandbox mode. This is because "the url returned to flytekit is a presigned url, with the host name which is kubernetes cluster local - like - http://minio.flyte.svc.cluster.local:9000" (@kumare3) and makes for a frustrating and broken experience when standing up flyte locally.

Goal: What should the final outcome look like, ideally?
Ideally the Get*ExecutionData endpoints [Workflow, Node, Task] should allow callers to specify whether they want populated inputs & outputs in addition to signed URL blobs pointing to where the same data is stored.

The endpoints should remain separate from Get*Execution as is, but offer more customizability.

Describe alternatives you've considered
Oneof responses that include either the URL blob or the populated inputs. However URL blobs are cheap to generate and lightweight to pass around so we can keep them as is.

Optionally the inputs & outputs can be returned inline in the Get*Execution responses but this risks bloating them.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline

Additional context
First raised by @kumare3

Is this a blocker for you to adopt Flyte
N/A

@katrogan katrogan added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Jul 20, 2020
@katrogan
Copy link
Contributor Author

cc @kumare3 @wild-endeavor

@katrogan katrogan removed the untriaged This issues has not yet been looked at by the Maintainers label Aug 12, 2020
@katrogan katrogan self-assigned this Aug 12, 2020
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
* change label selector op

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* lint error

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* test error

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* revert para name and keep old function

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* remove strconv.Itoa

Signed-off-by: aeioulisa <[email protected]>

* two lebal

Signed-off-by: aeioulisa <[email protected]>

* two lebal

Signed-off-by: aeioulisa <[email protected]>

* reset config and add split function

Signed-off-by: aeioulisa <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
* add flytescheduler to the release archives

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update flytepropeller version

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
Embedding a video to outline the Pytorch training process for single-node single GPUs.
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
* feat: webpack config split
* chore: import react from cdn
* chore: update readmes
* chore: add script to start prod build locally

Signed-off-by: Carina Ursu <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
* change label selector op

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* lint error

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* test error

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* revert para name and keep old function

Signed-off-by: Lisa <[email protected]>
Signed-off-by: aeioulisa <[email protected]>

* remove strconv.Itoa

Signed-off-by: aeioulisa <[email protected]>

* two lebal

Signed-off-by: aeioulisa <[email protected]>

* two lebal

Signed-off-by: aeioulisa <[email protected]>

* reset config and add split function

Signed-off-by: aeioulisa <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
* add flytescheduler to the release archives

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update flytepropeller version

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit that referenced this issue Sep 26, 2023
* add field

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pass task execution metadata from agent (#422)

* Pass task execution metadata from agent

Signed-off-by: Hongxin Liang <[email protected]>

* Add doc

Signed-off-by: Hongxin Liang <[email protected]>

* Update protos/flyteidl/admin/agent.proto

Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Honnix <[email protected]>

* Regenerate

---------

Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Honnix <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add tags to execution spec (#414)

* add tags to execution spec

Signed-off-by: Kevin Su <[email protected]>

* add tags to execution spec

Signed-off-by: Kevin Su <[email protected]>

* add comment

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Correct comment for array job max parallelism (#431)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add the scalar to the operand (#427)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add selector

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* move selectors from container to task metadata

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* drop only_preferred

Signed-off-by: Jeev B <[email protected]>

* Updating boilerplate to lock golangci-lint version (#435)

Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add unpartitioned selector

Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* fix oneof names

Signed-off-by: Jeev B <[email protected]>

* add build.os for read the docs

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Honnix <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: Honnix <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
eapolinario pushed a commit that referenced this issue Sep 26, 2023
* Revert "GPU Type (#419)"

This reverts commit 7bd98a9.

Signed-off-by: Jeev B <[email protected]>

* Restore .readthedocs.yml

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Jeev B <[email protected]>
pvditt pushed a commit that referenced this issue Dec 29, 2023
* add field

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pass task execution metadata from agent (#422)

* Pass task execution metadata from agent

Signed-off-by: Hongxin Liang <[email protected]>

* Add doc

Signed-off-by: Hongxin Liang <[email protected]>

* Update protos/flyteidl/admin/agent.proto

Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Honnix <[email protected]>

* Regenerate

---------

Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Honnix <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add tags to execution spec (#414)

* add tags to execution spec

Signed-off-by: Kevin Su <[email protected]>

* add tags to execution spec

Signed-off-by: Kevin Su <[email protected]>

* add comment

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Correct comment for array job max parallelism (#431)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add the scalar to the operand (#427)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add selector

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* move selectors from container to task metadata

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* drop only_preferred

Signed-off-by: Jeev B <[email protected]>

* Updating boilerplate to lock golangci-lint version (#435)

Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add unpartitioned selector

Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* fix oneof names

Signed-off-by: Jeev B <[email protected]>

* add build.os for read the docs

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Hongxin Liang <[email protected]>
Signed-off-by: Honnix <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: Honnix <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
pvditt pushed a commit that referenced this issue Dec 29, 2023
* Revert "GPU Type (#419)"

This reverts commit b92acad.

Signed-off-by: Jeev B <[email protected]>

* Restore .readthedocs.yml

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Jeev B <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <[email protected]>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <[email protected]>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <[email protected]>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <[email protected]>

* Updated tests.

Signed-off-by: Kamal Eybov <[email protected]>

* More tests.

Signed-off-by: Kamal Eybov <[email protected]>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Kamal Eybov <[email protected]>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <[email protected]>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <[email protected]>

---------

Signed-off-by: Kamal Eybov <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <[email protected]>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <[email protected]>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <[email protected]>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <[email protected]>

* Updated tests.

Signed-off-by: Kamal Eybov <[email protected]>

* More tests.

Signed-off-by: Kamal Eybov <[email protected]>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Kamal Eybov <[email protected]>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <[email protected]>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <[email protected]>

---------

Signed-off-by: Kamal Eybov <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <[email protected]>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <[email protected]>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <[email protected]>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <[email protected]>

* Updated tests.

Signed-off-by: Kamal Eybov <[email protected]>

* More tests.

Signed-off-by: Kamal Eybov <[email protected]>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Kamal Eybov <[email protected]>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <[email protected]>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <[email protected]>

---------

Signed-off-by: Kamal Eybov <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <[email protected]>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <[email protected]>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <[email protected]>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <[email protected]>

* Updated tests.

Signed-off-by: Kamal Eybov <[email protected]>

* More tests.

Signed-off-by: Kamal Eybov <[email protected]>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Kamal Eybov <[email protected]>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <[email protected]>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <[email protected]>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <[email protected]>

---------

Signed-off-by: Kamal Eybov <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant