Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-1660] Cache available workers and only count the available workers device free capacity #2827

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

turboFei
Copy link
Member

@turboFei turboFei commented Oct 18, 2024

What changes were proposed in this pull request?

  1. cache the available workers
  2. Only count the available workers device free capacity.
  3. place the metrics_AvailableWorkerCount_Value in overall and metrics_WorkerCount_Value in Master part

Why are the changes needed?

Cache the available workers to reduce the computation that need to loop the workers frequently.
To have an accurate device capacity overview that does not include the excluded workers, decommissioning workers, etc.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT.

image

@turboFei turboFei changed the title Only count the available workers device capacity [CELEBORN-1660] Only count the available workers device capacity Oct 18, 2024
@turboFei turboFei changed the title [CELEBORN-1660] Only count the available workers device capacity [CELEBORN-1660] Only count the available workers device free capacity Oct 19, 2024
@turboFei turboFei requested review from FMX and RexXiong October 19, 2024 02:44
@RexXiong
Copy link
Contributor

I think we can cache available workers(we can manually update the cache when occur exclude/shutdown..), the current process of computing the list of available workers can be resource-intensive, especially in large clusters.

@turboFei
Copy link
Member Author

I think we can cache available workers(we can manually update the cache when occur exclude/shutdown..), the current process of computing the list of available workers can be resource-intensive, especially in large clusters.

Before this, how about using map instead set for workers?
#2839
cc @RexXiong

@turboFei turboFei force-pushed the device_free branch 2 times, most recently from b5f209b to ebd53c5 Compare November 1, 2024 08:35
@turboFei turboFei marked this pull request as draft November 1, 2024 08:58
@turboFei turboFei marked this pull request as ready for review November 1, 2024 09:07
@turboFei turboFei marked this pull request as draft November 1, 2024 09:16
@turboFei turboFei marked this pull request as ready for review November 1, 2024 18:20
@turboFei turboFei force-pushed the device_free branch 2 times, most recently from 1563458 to 4a1202c Compare November 1, 2024 18:23
@turboFei turboFei changed the title [CELEBORN-1660] Only count the available workers device free capacity [CELEBORN-1660] Cache available workers and only count the available workers device free capacity Nov 1, 2024
@turboFei
Copy link
Member Author

turboFei commented Nov 4, 2024

This PR is ready. cc @FMX @RexXiong @zaynt4606

Copy link
Contributor

@zaynt4606 zaynt4606 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, #2856 can be covered by this PR.

@turboFei
Copy link
Member Author

turboFei commented Nov 6, 2024

will rebase the code

save

check

ut

doc

handle app heart beat

UT

UT

master metrics description
@turboFei
Copy link
Member Author

turboFei commented Nov 7, 2024

Hi @FMX and @RexXiong by any chance to help review this PR, thanks very much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants