Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concise dynamic node ids and names #705

Merged
merged 4 commits into from
Oct 15, 2021
Merged

Concise dynamic node ids and names #705

merged 4 commits into from
Oct 15, 2021

Conversation

kumare3
Copy link
Contributor

@kumare3 kumare3 commented Oct 14, 2021

TL;DR

PART 1:

  • Dynamic node ids currently have additional context, namely
    a shortened name of the dynamic task function
  • This causes a tremendous bloat and is completely dependent on the
    name of the task
  • This PR standardizes the name and affixes a simple d in the
    begining

PART 2:
Reduce node names, which are purely for decorative purposes

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Tracking Issue

fixes flyteorg/flyte#1660

Follow-up issue

NA

PART 1:
 - Dynamic node ids currently have additional context, namely
a shortened name of the dynamic task function
 - This causes a tremendous bloat and is completely dependent on the
name of the task
 - This PR standardizes the name and affixes a simple `d` in the
begining

PART 2:
Reduce node names, which are purely for decorative purposes

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Ketan Umare <[email protected]>
@kumare3 kumare3 changed the title Node id dynamic Concise dynamic node ids and names Oct 14, 2021
@kumare3 kumare3 requested a review from eapolinario October 14, 2021 23:35
Signed-off-by: Ketan Umare <[email protected]>
EngHabu
EngHabu previously approved these changes Oct 15, 2021
@codecov
Copy link

codecov bot commented Oct 15, 2021

Codecov Report

Merging #705 (c777360) into master (3b7c263) will increase coverage by 0.01%.
The diff coverage is 90.90%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #705      +/-   ##
==========================================
+ Coverage   85.70%   85.72%   +0.01%     
==========================================
  Files         355      356       +1     
  Lines       29703    29717      +14     
  Branches     2426     2428       +2     
==========================================
+ Hits        25457    25474      +17     
+ Misses       3605     3603       -2     
+ Partials      641      640       -1     
Impacted Files Coverage Δ
flytekit/core/base_task.py 89.04% <ø> (ø)
flytekit/core/python_function_task.py 84.61% <50.00%> (ø)
flytekit/core/workflow.py 89.36% <50.00%> (-0.31%) ⬇️
flytekit/common/utils.py 83.84% <100.00%> (+3.07%) ⬆️
flytekit/core/promise.py 78.00% <100.00%> (+0.29%) ⬆️
flytekit/core/reference_entity.py 91.85% <100.00%> (ø)
tests/flytekit/unit/common_tests/test_promise.py 100.00% <100.00%> (ø)
tests/flytekit/unit/common_tests/test_utils.py 100.00% <100.00%> (ø)
tests/flytekit/unit/core/test_workflows.py 98.79% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b7c263...c777360. Read the comment docs.

Signed-off-by: Ketan Umare <[email protected]>
@wild-endeavor
Copy link
Contributor

This won't work in certain dynamic cases if users are on the old version of Admin right? before the proper node to node relationship was done. Should we call that out?

@eapolinario
Copy link
Collaborator

This won't work in certain dynamic cases if users are on the old version of Admin right? before the proper node to node relationship was done. Should we call that out?

This changed with flyteorg/flyteadmin#111 ?

@kumare3
Copy link
Contributor Author

kumare3 commented Oct 15, 2021

@wild-endeavor we have defaulted to node to node versions now. so it is the default

Comment on lines +738 to +739
if name is None:
return ""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we shouldn't need this, right?

@kumare3 kumare3 merged commit 1e5c9d5 into master Oct 15, 2021
aeioulisa pushed a commit to aeioulisa/flytekit that referenced this pull request Oct 19, 2021
* Concise: dynamic node ids and names

PART 1:
 - Dynamic node ids currently have additional context, namely
a shortened name of the dynamic task function
 - This causes a tremendous bloat and is completely dependent on the
name of the task
 - This PR standardizes the name and affixes a simple `d` in the
begining

PART 2:
Reduce node names, which are purely for decorative purposes

Signed-off-by: Ketan Umare <[email protected]>

* unit tests

Signed-off-by: Ketan Umare <[email protected]>

* fix test

Signed-off-by: Ketan Umare <[email protected]>

* unit tests

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Lisa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core][Scale] Large number of launchplans yielded (8k+) in dynamic workflows cause memory bloat
4 participants