Add listing to stow storage apis #5674

bgedik · 2024-08-20T07:35:09Z

Tracking issue

Part of RFC https://github.com/flyteorg/flyte/blob/b3add894fd879d6b6c11e37ba292ae04bf60afeb/rfc/system/5598-deterministic-errors-distributed-training.md#flytepropeller-backend

Why are the changes needed?

Deterministic error propagation, see: https://github.com/flyteorg/flyte/blob/b3add894fd879d6b6c11e37ba292ae04bf60afeb/rfc/system/5598-deterministic-errors-distributed-training.md#flytepropeller-backend

What changes were proposed in this pull request?

Adding listing APIs

How was this patch tested?

Unit tests

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

#5616

Docs link

N/A

Signed-off-by: Bugra Gedik <[email protected]>

* Refactor panic handling to middleware Signed-off-by: Jason Parraga <[email protected]> * Remove registration of old panicCounter Signed-off-by: Jason Parraga <[email protected]> * Add test coverage Signed-off-by: Jason Parraga <[email protected]> --------- Signed-off-by: Jason Parraga <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* TEST build Signed-off-by: Future-Outlier <[email protected]> * remove emphasize-lines Signed-off-by: Future-Outlier <[email protected]> * test build Signed-off-by: Future-Outlier <[email protected]> * revert Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…teorg#5612) * FlytePropeller Compiler Avoid Crash when Type not found Signed-off-by: Future-Outlier <[email protected]> * Update pingsu's error message advices Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: pingsutw <[email protected]> * fix lint Signed-off-by: Future-Outlier <[email protected]> * Trigger CI Signed-off-by: Future-Outlier <[email protected]> * Trigger CI Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: pingsutw <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

…#5623) Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

* first version Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: ddl-rliu <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* add send arg Signed-off-by: Yee Hing Tong <[email protected]> * Add acction to remove cache in gh runner Signed-off-by: Eduardo Apolinario <[email protected]> * Use correct checked out path Signed-off-by: Eduardo Apolinario <[email protected]> * Path in strings Signed-off-by: Eduardo Apolinario <[email protected]> * Checkout repo in root Signed-off-by: Eduardo Apolinario <[email protected]> * Use the correct path to new action Signed-off-by: Eduardo Apolinario <[email protected]> * Do not use gh var in path to clear-action-cache Signed-off-by: Eduardo Apolinario <[email protected]> * Remove wrong invocation of clear-action-cache Signed-off-by: Eduardo Apolinario <[email protected]> * GITHUB_WORKSPACE is implicit in the checkout action Signed-off-by: Eduardo Apolinario <[email protected]> * Refer to local `flyte` directory Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…g#5635) * Make flyteidl releases go through a manual gh workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Make flytectl releases go through a manual gh workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Rewrite the documentation for `version` and clarify wording in RELEASE.md Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* fix CHANGELOG-v0.2.0.md Signed-off-by: Christina <[email protected]> * fix CHANGELOG-v1.0.2-b1.md Signed-off-by: Christina <[email protected]> * fix CHANGELOG-v1.1.0.md Signed-off-by: Christina <[email protected]> * fix CHANGELOG-v1.3.0.md Signed-off-by: Christina <[email protected]> --------- Signed-off-by: Christina <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

* Fetch all tags in flyteidl-release.yml Signed-off-by: Eduardo Apolinario <[email protected]> * Fix sed expression for npm job Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…g#5545) * update Signed-off-by: Desi Hsu <[email protected]> * dco Signed-off-by: Desi Hsu <[email protected]> * dco Signed-off-by: Desi Hsu <[email protected]> * typo Signed-off-by: Desi Hsu <[email protected]> --------- Signed-off-by: Desi Hsu <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

flyteorg#5648) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

flyteorg#5649) * Don't error when attempting to trigger schedules for inactive projects Signed-off-by: Katrina Rogan <[email protected]> * regen Signed-off-by: Katrina Rogan <[email protected]> --------- Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

….gedik/multi_pod_task_failure_handling_env

Signed-off-by: Bugra Gedik <[email protected]>

codecov · 2024-08-20T17:22:15Z

Codecov Report

Attention: Patch coverage is 43.42105% with 43 lines in your changes missing coverage. Please review.

Project coverage is 36.22%. Comparing base (86c63f7) to head (16454bf).
Report is 158 commits behind head on master.

Files with missing lines	Patch %	Lines
...ytestdlib/storage/mocks/composed_protobuf_store.go	0.00%	27 Missing ⚠️
flytestdlib/storage/stow_store.go	67.74%	6 Missing and 4 partials ⚠️
flyteadmin/pkg/common/mocks/storage.go	0.00%	2 Missing ⚠️
flytepropeller/pkg/utils/failing_datastore.go	0.00%	2 Missing ⚠️
flytestdlib/storage/mem_store.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5674   +/-   ##
=======================================
  Coverage   36.21%   36.22%           
=======================================
  Files        1303     1303           
  Lines      109568   109644   +76     
=======================================
+ Hits        39683    39715   +32     
- Misses      65765    65805   +40     
- Partials     4120     4124    +4

Flag	Coverage Δ
unittests-datacatalog	`51.37% <ø> (ø)`
unittests-flyteadmin	`55.63% <0.00%> (-0.01%)`	⬇️
unittests-flytecopilot	`12.17% <ø> (ø)`
unittests-flytectl	`62.21% <ø> (ø)`
unittests-flyteidl	`7.12% <ø> (ø)`
unittests-flyteplugins	`53.35% <ø> (ø)`
unittests-flytepropeller	`41.75% <0.00%> (-0.01%)`	⬇️
unittests-flytestdlib	`55.19% <45.83%> (-0.17%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* More concise definition of launchplan Signed-off-by: Eduardo Apolinario <[email protected]> * Update docs/flyte_fundamentals/tasks_workflows_and_launch_plans.md Co-authored-by: Nikki Everett <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> * Update docs/flyte_fundamentals/tasks_workflows_and_launch_plans.md Co-authored-by: Nikki Everett <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Nikki Everett <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* save values Signed-off-by: Yee Hing Tong <[email protected]> * move things up Signed-off-by: Yee Hing Tong <[email protected]> * tests Signed-off-by: Yee Hing Tong <[email protected]> * unit test Signed-off-by: Yee Hing Tong <[email protected]> * imports for client test Signed-off-by: Yee Hing Tong <[email protected]> * more test Signed-off-by: Yee Hing Tong <[email protected]> * don't test admin connection Signed-off-by: Yee Hing Tong <[email protected]> * disable client for config Signed-off-by: Yee Hing Tong <[email protected]> * make generate Signed-off-by: Yee Hing Tong <[email protected]> * hide behind a once Signed-off-by: Yee Hing Tong <[email protected]> * typo Signed-off-by: Yee Hing Tong <[email protected]> * reset client builder test Signed-off-by: Yee Hing Tong <[email protected]> * reset client test Signed-off-by: Yee Hing Tong <[email protected]> * revert propeller Signed-off-by: Yee Hing Tong <[email protected]> * delay invocation even further Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* Update Flyte Components Signed-off-by: Flyte-Bot <[email protected]> * Add changelog and bump version in conf.py Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Flyte-Bot <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: eapolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* Update Flyte Components Signed-off-by: Flyte-Bot <[email protected]> * Bump version in conf.py and add changelog Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Flyte-Bot <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: eapolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…5704) * Do not emit execution id label by default in single binary Signed-off-by: Eduardo Apolinario <[email protected]> * Update flytestdlib/contextutils/context.go Co-authored-by: Thomas J. Fan <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…lyteorg#5705) Signed-off-by: Bugra Gedik <[email protected]>

* wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * update tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * address comment Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Future-Outlier <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * fix ci Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* add echo tasks Signed-off-by: Future-Outlier <[email protected]> * update nikki's advice Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: nikki everett <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: nikki everett <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: wayner0628 <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…ent (flyteorg#5709) Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: wayner0628 <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* Update helm/docs per changes in supported task discovery Signed-off-by: Jason Parraga <[email protected]> * Cleanup flyte-binary Signed-off-by: Jason Parraga <[email protected]> * fixes Signed-off-by: Jason Parraga <[email protected]> --------- Signed-off-by: Jason Parraga <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* Add Logging for Agent-Supported Task Types Signed-off-by: Future-Outlier <[email protected]> * use make(map[string]struct{}) Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* apply pod config to init containers Signed-off-by: Samhita Alla <[email protected]> * remove container Signed-off-by: Samhita Alla <[email protected]> * add test Signed-off-by: Samhita Alla <[email protected]> * Add bool to check init containers Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

* update langauge Signed-off-by: nikki everett <[email protected]> * Update docs/quickstart_guide.md Co-authored-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Nikki Everett <[email protected]> --------- Signed-off-by: nikki everett <[email protected]> Signed-off-by: Nikki Everett <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

* feat: Use deterministic execution names in scheduler Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…tation (flyteorg#5731) Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

…lyteorg#5716) * Fix flytectl returning the oldest workflow when using --latest flag (flyteorg#5716) Signed-off-by: Rafael Raposo <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

- flyteorg#5032 updated builds to use Go 1.22 -- but in practice it looks like flytectl is being built with an outdated toolchain based on Go 1.22.0 instead of Go 1.22.5 or 1.22.6 (i.e. latest) See https://go.dev/doc/toolchain for further details For instance, flytectl v0.9.0 reports being built with Go 1.22.0 ❯ go version -m -v flytectl flytectl: go1.22.0 Ensure the proper Go environment for security reasons Signed-off-by: ddl-ebrown <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Signed-off-by: Bugra Gedik <[email protected]>

…bgedik/flyte into bugra.gedik/add_listing_to_storage_apis

bgedik · 2024-09-12T05:13:22Z

Messed up the DCO pretty bad, will create a new one

bgedik · 2024-09-12T05:29:36Z

#5741

bgedik and others added 29 commits August 15, 2024 19:41

Add environment variable for pod name

7306683

Signed-off-by: Bugra Gedik <[email protected]>

Fix nil pointer when task plugin load returns error (flyteorg#5622)

91d6d40

Signed-off-by: Bugra Gedik <[email protected]>

Log stack trace when refresh cache sync recovers from panic (flyteorg…

eae4bf5

…#5623) Signed-off-by: Bugra Gedik <[email protected]>

use private-key (flyteorg#5626)

dcbc55a

Signed-off-by: Bugra Gedik <[email protected]>

Explain how Agent Secret Works (flyteorg#5625)

61cffe8

* first version Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Fix typo in execution manager (flyteorg#5619)

2cf52ef

Signed-off-by: ddl-rliu <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

document the process of setting ttl for a ray cluster (flyteorg#5636)

fe9fb67

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Add CustomHeaderMatcher to pass additional headers (flyteorg#5563)

0514430

Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Use enable_deck=True in docs (flyteorg#5645)

a9beb65

Signed-off-by: Bugra Gedik <[email protected]>

Install pyarrow in sandbox functional tests (flyteorg#5647)

ee4783e

Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Correct "sucessfile" to "successfile" (flyteorg#5652)

aff319b

Signed-off-by: Bugra Gedik <[email protected]>

Fix ordering for custom template values in cluster resource controller (

4929522

flyteorg#5648) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

fix tests

2ddb4d2

Signed-off-by: Bugra Gedik <[email protected]>

Merge branch 'master' of https://github.com/flyteorg/flyte into bugra…

4e7a1c5

….gedik/multi_pod_task_failure_handling_env

change to shorter names

1b24ca2

Signed-off-by: Bugra Gedik <[email protected]>

change to shorter names

ede7536

Signed-off-by: Bugra Gedik <[email protected]>

change to shorter names

7a14649

Signed-off-by: Bugra Gedik <[email protected]>

change to shorter names

783d75e

Signed-off-by: Bugra Gedik <[email protected]>

change to shorter names

c89950f

Signed-off-by: Bugra Gedik <[email protected]>

Add list operation to stow container

8215edc

Signed-off-by: Bugra Gedik <[email protected]>

eapolinario and others added 27 commits September 12, 2024 04:57

[flytectl] DataConfig missing from TaskSpec (flyteorg#5692)

3b690d2

Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Enable echo plugin by default (flyteorg#5679)

add4e48

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Using new offloaded metadata literal message for literal offloading (f…

e126da1

…lyteorg#5705) Signed-off-by: Bugra Gedik <[email protected]>

Improve execution name readability (flyteorg#5637)

c5f7d84

Signed-off-by: wayner0628 <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Configure imagePullPolicy to be Always pull on flyte sandbox environm…

b46e203

…ent (flyteorg#5709) Signed-off-by: Bugra Gedik <[email protected]>

should not set echo plugin as default (flyteorg#5713)

2afe101

Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Move default execution name generation to flyteadmin (flyteorg#5714)

9f630b8

Signed-off-by: wayner0628 <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Refactor flyteadmin to pass proto structs as pointers (flyteorg#5717)

8c303f4

Signed-off-by: Bugra Gedik <[email protected]>

Enable in every helm chart (flyteorg#5730)

daaff18

Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

[Docs][flyteagent] Remove agent-service config in flyte agent documen…

d3a3fd1

…tation (flyteorg#5731) Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

Fix flytectl returning the oldest workflow when using --latest flag (f…

f30613b

…lyteorg#5716) * Fix flytectl returning the oldest workflow when using --latest flag (flyteorg#5716) Signed-off-by: Rafael Raposo <[email protected]> Signed-off-by: Bugra Gedik <[email protected]>

mockery

cbdddd4

Signed-off-by: Bugra Gedik <[email protected]>

mockery

31ba83d

Signed-off-by: Bugra Gedik <[email protected]>

Merge branch 'bugra.gedik/add_listing_to_storage_apis' of github.com:…

16454bf

…bgedik/flyte into bugra.gedik/add_listing_to_storage_apis

bgedik closed this Sep 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add listing to stow storage apis #5674

Add listing to stow storage apis #5674

bgedik commented Aug 20, 2024

codecov bot commented Aug 20, 2024 •

edited

Loading

bgedik commented Sep 12, 2024

bgedik commented Sep 12, 2024

Add listing to stow storage apis #5674

Add listing to stow storage apis #5674

Conversation

bgedik commented Aug 20, 2024

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

codecov bot commented Aug 20, 2024 • edited Loading

Codecov Report

bgedik commented Sep 12, 2024

bgedik commented Sep 12, 2024

codecov bot commented Aug 20, 2024 •

edited

Loading