Skip to content
This repository has been archived by the owner on Oct 23, 2023. It is now read-only.

Add Reasons field to TaskExecutionClosure to track time-series of reasons #382

Merged
merged 3 commits into from
Mar 21, 2023

Conversation

hamersaw
Copy link
Contributor

@hamersaw hamersaw commented Mar 14, 2023

TL;DR

This PR adds a Reasons field to the TaskExecutionClosure. This field tracks a timeseries of messages, annotated with the timestamp with which they occurred. This is useful for better visualizations of task execution.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

flyteorg/flyte#3501

Follow-up issue

NA

@codecov
Copy link

codecov bot commented Mar 14, 2023

Codecov Report

Merging #382 (e4cc141) into master (e1a667e) will increase coverage by 2.33%.
The diff coverage is n/a.

❗ Current head e4cc141 differs from pull request most recent head c2c2727. Consider uploading reports for the commit c2c2727 to get more accurate results

@@            Coverage Diff             @@
##           master     #382      +/-   ##
==========================================
+ Coverage   76.71%   79.05%   +2.33%     
==========================================
  Files          18       18              
  Lines        1426     1227     -199     
==========================================
- Hits         1094      970     -124     
+ Misses        280      205      -75     
  Partials       52       52              
Flag Coverage Δ
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 18 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@hamersaw hamersaw marked this pull request as ready for review March 14, 2023 19:01
@hamersaw hamersaw merged commit 0f18851 into master Mar 21, 2023
@hamersaw hamersaw deleted the feature/reason-time-series branch March 21, 2023 07:15
eapolinario pushed a commit that referenced this pull request May 16, 2023
…sons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <[email protected]>

* added docs

Signed-off-by: Daniel Rammer <[email protected]>

* actually finishing docs too

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
eapolinario added a commit that referenced this pull request May 16, 2023
* added dynamic_job_spec_uri to dynamic workflow metadata and node execution closure (#360)

Signed-off-by: Daniel Rammer <[email protected]>

* Use TokenCache in ClientCredentialsTokenSourceProvider (#377)

* Init customTokenSource.refreshTime (#381)

Signed-off-by: Andrew Dye <[email protected]>

* added DataLoadingConfig to K8sPod message (#368)

Signed-off-by: Daniel Rammer <[email protected]>

* Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <[email protected]>

* added docs

Signed-off-by: Daniel Rammer <[email protected]>

* actually finishing docs too

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>

* Create service for runtime metrics (#367)

* added span messages

Signed-off-by: Daniel Rammer <[email protected]>

* added endpoints to service

Signed-off-by: Daniel Rammer <[email protected]>

* generated mocks

Signed-off-by: Daniel Rammer <[email protected]>

* removed get task execution metrics rpc

Signed-off-by: Daniel Rammer <[email protected]>

* added EXECUTION_IDLE category

Signed-off-by: Daniel Rammer <[email protected]>

* updated PLUGIN_EXECUTION to PLUGIN_RUNTIME

Signed-off-by: Daniel Rammer <[email protected]>

* removed recorded_at on workflow and node level events

Signed-off-by: Daniel Rammer <[email protected]>

* added docs for task event reported_at field

Signed-off-by: Daniel Rammer <[email protected]>

* removed GetNodeExecutionMetrics endpoint - will implement later if necessary

Signed-off-by: Daniel Rammer <[email protected]>

* updated docs

Signed-off-by: Daniel Rammer <[email protected]>

* added reported_at for node execution events

Signed-off-by: Daniel Rammer <[email protected]>

* fixed typo

Signed-off-by: Daniel Rammer <[email protected]>

* fixed typos and removed dead code

Signed-off-by: Daniel Rammer <[email protected]>

* updated categories

Signed-off-by: Daniel Rammer <[email protected]>

* added workflow setup and teardown categories

Signed-off-by: Daniel Rammer <[email protected]>

* simplified span message and moved to flyteidl.core

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>

* Remove misleading token refresh logic from client credentials token source provider (#383)

* Out of core plugin (#378)

* Add backend plugin system service

Signed-off-by: Kevin Su <[email protected]>

* Add backend plugin system service

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update state

Signed-off-by: Kevin Su <[email protected]>

* update state

Signed-off-by: Kevin Su <[email protected]>

* dics

Signed-off-by: Kevin Su <[email protected]>

* Remove output prefix from get request

Signed-off-by: Kevin Su <[email protected]>

* update

Signed-off-by: Kevin Su <[email protected]>

* remove prev state

Signed-off-by: Kevin Su <[email protected]>

* update proto

Signed-off-by: Kevin Su <[email protected]>

* remove error message

Signed-off-by: Kevin Su <[email protected]>

* update comment

Signed-off-by: Kevin Su <[email protected]>

* make generate

Signed-off-by: Kevin Su <[email protected]>

* Rename the service

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>

* Feat: Add `ElasticConfig` message type for torch elastic training (#394)

* Add elastic config args to pytorch proto

Signed-off-by: Fabio Graetz <[email protected]>

* Add elastic config message type for torchrun training

Signed-off-by: Fabio Graetz <[email protected]>

---------

Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>

* Retract 1.4.x (#397)

Signed-off-by: eduardo apolinario <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>

* Data addresses #minor (#391)

Signed-off-by: Yee Hing Tong <[email protected]>

* Refactor kf-operator plugins configs and support setting different specs for different replica groups (#386)

* refactor kubeflow operators proto

Signed-off-by: Yubo Wang <[email protected]>

* add back the original proto for backward compatible

Signed-off-by: Yubo Wang <[email protected]>

* clean up comments

Signed-off-by: Yubo Wang <[email protected]>

* add kubeflow.rs

Signed-off-by: Yubo Wang <[email protected]>

* add elastic config

Signed-off-by: Yubo Wang <[email protected]>

* add command to MPI

Signed-off-by: Yubo Wang <[email protected]>

* add slots and command to mpi spec

Signed-off-by: Yubo Wang <[email protected]>

---------

Signed-off-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>

* add user_identifier (#388)

Signed-off-by: byhsu <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Co-authored-by: byhsu <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>

* Add envs to execution spec (#400)

Signed-off-by: Kevin Su <[email protected]>

* Support union and none type in flyteidl (#401)

* add support for Union Scalar

Signed-off-by: Yubo Wang <[email protected]>

* support union type and literals

Signed-off-by: Yubo Wang <[email protected]>

* change union type extraction

Signed-off-by: Yubo Wang <[email protected]>

---------

Signed-off-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: Kevin Su <[email protected]>

* Rename user_identity to execution_identity (#402)

Signed-off-by: byhsu <[email protected]>
Co-authored-by: byhsu <[email protected]>

* make generate

Signed-off-by: eduardo apolinario <[email protected]>

* Revert "Support union and none type in flyteidl (#401)"

This reverts commit 3284f61.

Signed-off-by: Eduardo Apolinario <[email protected]>

* We should not update flyteidl version in backend components in the case of this branch

Signed-off-by: eduardo apolinario <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Andrew Dye <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yubo Wang <[email protected]>
Signed-off-by: byhsu <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
Co-authored-by: Andrew Dye <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: ByronHsu <[email protected]>
Co-authored-by: byhsu <[email protected]>
eapolinario pushed a commit that referenced this pull request Jun 27, 2023
…sons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <[email protected]>

* added docs

Signed-off-by: Daniel Rammer <[email protected]>

* actually finishing docs too

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
eapolinario added a commit that referenced this pull request Jun 28, 2023
* Adding support for structured dataset (#369)

Signed-off-by: pmahindrakar-oss <[email protected]>

* added dynamic_job_spec_uri to dynamic workflow metadata and node execution closure (#360)

Signed-off-by: Daniel Rammer <[email protected]>

* Use TokenCache in ClientCredentialsTokenSourceProvider (#377)

* Init customTokenSource.refreshTime (#381)

Signed-off-by: Andrew Dye <[email protected]>

* added DataLoadingConfig to K8sPod message (#368)

Signed-off-by: Daniel Rammer <[email protected]>

* Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <[email protected]>

* added docs

Signed-off-by: Daniel Rammer <[email protected]>

* actually finishing docs too

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>

* Create service for runtime metrics (#367)

* added span messages

Signed-off-by: Daniel Rammer <[email protected]>

* added endpoints to service

Signed-off-by: Daniel Rammer <[email protected]>

* generated mocks

Signed-off-by: Daniel Rammer <[email protected]>

* removed get task execution metrics rpc

Signed-off-by: Daniel Rammer <[email protected]>

* added EXECUTION_IDLE category

Signed-off-by: Daniel Rammer <[email protected]>

* updated PLUGIN_EXECUTION to PLUGIN_RUNTIME

Signed-off-by: Daniel Rammer <[email protected]>

* removed recorded_at on workflow and node level events

Signed-off-by: Daniel Rammer <[email protected]>

* added docs for task event reported_at field

Signed-off-by: Daniel Rammer <[email protected]>

* removed GetNodeExecutionMetrics endpoint - will implement later if necessary

Signed-off-by: Daniel Rammer <[email protected]>

* updated docs

Signed-off-by: Daniel Rammer <[email protected]>

* added reported_at for node execution events

Signed-off-by: Daniel Rammer <[email protected]>

* fixed typo

Signed-off-by: Daniel Rammer <[email protected]>

* fixed typos and removed dead code

Signed-off-by: Daniel Rammer <[email protected]>

* updated categories

Signed-off-by: Daniel Rammer <[email protected]>

* added workflow setup and teardown categories

Signed-off-by: Daniel Rammer <[email protected]>

* simplified span message and moved to flyteidl.core

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>

* Remove misleading token refresh logic from client credentials token source provider (#383)

* Out of core plugin (#378)

* Add backend plugin system service

Signed-off-by: Kevin Su <[email protected]>

* Add backend plugin system service

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update state

Signed-off-by: Kevin Su <[email protected]>

* update state

Signed-off-by: Kevin Su <[email protected]>

* dics

Signed-off-by: Kevin Su <[email protected]>

* Remove output prefix from get request

Signed-off-by: Kevin Su <[email protected]>

* update

Signed-off-by: Kevin Su <[email protected]>

* remove prev state

Signed-off-by: Kevin Su <[email protected]>

* update proto

Signed-off-by: Kevin Su <[email protected]>

* remove error message

Signed-off-by: Kevin Su <[email protected]>

* update comment

Signed-off-by: Kevin Su <[email protected]>

* make generate

Signed-off-by: Kevin Su <[email protected]>

* Rename the service

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>

* Feat: Add `ElasticConfig` message type for torch elastic training (#394)

* Add elastic config args to pytorch proto

Signed-off-by: Fabio Graetz <[email protected]>

* Add elastic config message type for torchrun training

Signed-off-by: Fabio Graetz <[email protected]>

---------

Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>

* Retract 1.4.x (#397)

Signed-off-by: eduardo apolinario <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>

* Data addresses #minor (#391)

Signed-off-by: Yee Hing Tong <[email protected]>

* Refactor kf-operator plugins configs and support setting different specs for different replica groups (#386)

* refactor kubeflow operators proto

Signed-off-by: Yubo Wang <[email protected]>

* add back the original proto for backward compatible

Signed-off-by: Yubo Wang <[email protected]>

* clean up comments

Signed-off-by: Yubo Wang <[email protected]>

* add kubeflow.rs

Signed-off-by: Yubo Wang <[email protected]>

* add elastic config

Signed-off-by: Yubo Wang <[email protected]>

* add command to MPI

Signed-off-by: Yubo Wang <[email protected]>

* add slots and command to mpi spec

Signed-off-by: Yubo Wang <[email protected]>

---------

Signed-off-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>

* add user_identifier (#388)

Signed-off-by: byhsu <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Co-authored-by: byhsu <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>

* Add envs to execution spec (#400)

Signed-off-by: Kevin Su <[email protected]>

* Support union and none type in flyteidl (#401)

* add support for Union Scalar

Signed-off-by: Yubo Wang <[email protected]>

* support union type and literals

Signed-off-by: Yubo Wang <[email protected]>

* change union type extraction

Signed-off-by: Yubo Wang <[email protected]>

---------

Signed-off-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: Kevin Su <[email protected]>

* Rename user_identity to execution_identity (#402)

Signed-off-by: byhsu <[email protected]>
Co-authored-by: byhsu <[email protected]>

* Single literal in GetDataResponse (#404)

Signed-off-by: Yee Hing Tong <[email protected]>

* Add namespace to execution system metadata (#406)

Signed-off-by: Katrina Rogan <[email protected]>

* Add oauth2 http proxy client (#405)

Signed-off-by: byhsu <[email protected]>

* Rename externalPluginService to AgentService (#410)

* Rename externalPluginService to AgentService

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>

* Add external_plugin_service proto back to the idl (#413)

* Add external-plugin-service proto back to the idl

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* update idll

Signed-off-by: Kevin Su <[email protected]>

* update idll

Signed-off-by: Kevin Su <[email protected]>

* AsyncAgentService

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>

* Rerun make generate

Signed-off-by: eduardo apolinario <[email protected]>

---------

Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Andrew Dye <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yubo Wang <[email protected]>
Signed-off-by: byhsu <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Co-authored-by: pmahindrakar-oss <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
Co-authored-by: Andrew Dye <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>
Co-authored-by: eduardo apolinario <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: ByronHsu <[email protected]>
Co-authored-by: byhsu <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
eapolinario pushed a commit that referenced this pull request Sep 8, 2023
…sons (#382)

* added a time-series of reasons to the TaskExecution closure

Signed-off-by: Daniel Rammer <[email protected]>

* added docs

Signed-off-by: Daniel Rammer <[email protected]>

* actually finishing docs too

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants