Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update array node map task to support an additional plugin #2934

Merged
merged 11 commits into from
Dec 2, 2024

Conversation

pvditt
Copy link
Contributor

@pvditt pvditt commented Nov 15, 2024

Tracking issue

closes: https://linear.app/unionai/issue/COR-2239/add-additional-dataexecution-modes-for-array-node

Why are the changes needed?

start to add support for array node map task to run additional plugins (ie actors)

need to merge flyteorg/flyte#6018 in before merging this

What changes were proposed in this pull request?

  • set is_original_sub_node_interface to clean up array node compilation
  • set array node map task to utilize full state storage for actors
  • fix bug in get_custom that is used as part of the task template for the subtask (no need to set array node/parent level metadata for the subtask)

How was this patch tested?

  • added unit test

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Copy link

codecov bot commented Nov 15, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 46.65%. Comparing base (fdb7676) to head (6149e2b).

Files with missing lines Patch % Lines
flytekit/core/array_node_map_task.py 45.45% 6 Missing ⚠️
flytekit/core/array_node.py 66.66% 1 Missing ⚠️
flytekit/models/core/workflow.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2934      +/-   ##
==========================================
- Coverage   51.25%   46.65%   -4.60%     
==========================================
  Files         200      200              
  Lines       20835    20849      +14     
  Branches     2688     2689       +1     
==========================================
- Hits        10678     9728     -950     
- Misses       9559    10644    +1085     
+ Partials      598      477     -121     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pvditt pvditt marked this pull request as ready for review November 15, 2024 19:33
@pvditt pvditt changed the title Update array node map task Update array node map task to support an additional plugin Nov 15, 2024
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
This reverts commit e98bc6e.

Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Paul Dittamo <[email protected]>
@pvditt pvditt force-pushed the update-array-node-map-task branch from 86bfe79 to 505570a Compare November 21, 2024 01:21
pvditt and others added 2 commits November 25, 2024 16:10
@eapolinario eapolinario merged commit 209fa65 into master Dec 2, 2024
104 checks passed
arbaobao pushed a commit to arbaobao/flytekit that referenced this pull request Dec 4, 2024
…2934)

* have get_custom return subtask and remove un-needed array job

Signed-off-by: Paul Dittamo <[email protected]>

* set is_original_sub_node_interface + FULL_STATE for python function task extensions

Signed-off-by: Paul Dittamo <[email protected]>

* lint

Signed-off-by: Paul Dittamo <[email protected]>

* unit test

Signed-off-by: Paul Dittamo <[email protected]>

* clean up

Signed-off-by: Paul Dittamo <[email protected]>

* idl update

Signed-off-by: Paul Dittamo <[email protected]>

* Revert "idl update"

This reverts commit e98bc6e.

Signed-off-by: Paul Dittamo <[email protected]>

* pass in boolvalue

Signed-off-by: Paul Dittamo <[email protected]>

* update unit test

Signed-off-by: Paul Dittamo <[email protected]>

* Set flyteidl lower bound

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants