Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: Enable score based balance channel policy #38143

Merged

Conversation

weiliu1031
Copy link
Contributor

issue: #38142
current balance channel policy only consider current collection's distribution, so if all collections has 1 channel, and all channels has been loaded on same querynode, after querynode num increase, balance channel won't be triggered.

This PR enable score based balance channel policy, to achieve:

  1. distribute all channels evenly across multiple querynodes
  2. distribute each collection's channel evenly across multiple querynodes.

@sre-ci-robot sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Dec 2, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 2, 2024
@weiliu1031 weiliu1031 force-pushed the enable_scored_based_balance_channel branch from 50bd48a to e175a84 Compare December 2, 2024 11:40
Copy link
Contributor

mergify bot commented Dec 2, 2024

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Copy link

codecov bot commented Dec 2, 2024

Codecov Report

Attention: Patch coverage is 97.72727% with 5 lines in your changes missing coverage. Please review.

Project coverage is 81.06%. Comparing base (fec31fe) to head (d001eb8).
Report is 18 commits behind head on master.

Files with missing lines Patch % Lines
...ernal/querycoordv2/balance/score_based_balancer.go 98.42% 2 Missing and 1 partial ⚠️
...erycoordv2/balance/channel_level_score_balancer.go 66.66% 1 Missing ⚠️
internal/querycoordv2/utils/meta.go 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #38143      +/-   ##
==========================================
+ Coverage   81.04%   81.06%   +0.01%     
==========================================
  Files        1372     1372              
  Lines      191677   191875     +198     
==========================================
+ Hits       155346   155536     +190     
- Misses      30828    30829       +1     
- Partials     5503     5510       +7     
Components Coverage Δ
Client 74.58% <ø> (ø)
Core 68.83% <ø> (ø)
Go 83.17% <97.72%> (+0.01%) ⬆️
Files with missing lines Coverage Δ
internal/querycoordv2/balance/balance.go 94.11% <100.00%> (ø)
...al/querycoordv2/balance/rowcount_based_balancer.go 94.73% <100.00%> (-0.36%) ⬇️
internal/querycoordv2/checkers/channel_checker.go 87.13% <100.00%> (ø)
internal/querycoordv2/handlers.go 74.60% <100.00%> (+2.64%) ⬆️
pkg/util/paramtable/component_param.go 98.37% <100.00%> (+<0.01%) ⬆️
...erycoordv2/balance/channel_level_score_balancer.go 89.83% <66.66%> (ø)
internal/querycoordv2/utils/meta.go 71.22% <0.00%> (-2.88%) ⬇️
...ernal/querycoordv2/balance/score_based_balancer.go 98.72% <98.42%> (-0.26%) ⬇️

... and 35 files with indirect coverage changes

@weiliu1031
Copy link
Contributor Author

/run-cpu-e2e

Copy link
Contributor

mergify bot commented Dec 2, 2024

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@weiliu1031 weiliu1031 force-pushed the enable_scored_based_balance_channel branch from e175a84 to 8126aac Compare December 3, 2024 02:45
Copy link
Contributor

mergify bot commented Dec 3, 2024

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@weiliu1031
Copy link
Contributor Author

rerun go-sdk

Copy link
Contributor

mergify bot commented Dec 3, 2024

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@weiliu1031
Copy link
Contributor Author

rerun go-sdk

Copy link
Contributor

mergify bot commented Dec 3, 2024

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@weiliu1031
Copy link
Contributor Author

rerun go-sdk

@mergify mergify bot added the ci-passed label Dec 3, 2024
czs007 pushed a commit that referenced this pull request Dec 9, 2024
issue: #38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:

1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

pr: #38143

---------

Signed-off-by: bigsheeper <[email protected]>
Co-authored-by: Wei Liu <[email protected]>
Copy link
Collaborator

@xiaofan-luan xiaofan-luan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: weiliu1031, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

current balance channel policy only consider current collection's distribution,
so if all collections has 1 channel, and all channels has been loaded on
same querynode, after querynode num increase, balance channel won't be
triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
   querynodes.

Signed-off-by: Wei Liu <[email protected]>
Signed-off-by: Wei Liu <[email protected]>
@weiliu1031 weiliu1031 force-pushed the enable_scored_based_balance_channel branch from 8126aac to d001eb8 Compare December 10, 2024 09:54
@mergify mergify bot added ci-passed and removed ci-passed labels Dec 10, 2024
@xiaofan-luan
Copy link
Collaborator

/lgtm

@sre-ci-robot sre-ci-robot merged commit e279ccf into milvus-io:master Dec 11, 2024
20 checks passed
weiliu1031 added a commit to weiliu1031/milvus that referenced this pull request Dec 11, 2024
issue: milvus-io#38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

---------

Signed-off-by: Wei Liu <[email protected]>
weiliu1031 added a commit to weiliu1031/milvus that referenced this pull request Dec 11, 2024
issue: milvus-io#38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

---------

Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot pushed a commit that referenced this pull request Dec 13, 2024
issue: #38142
pr: #38143
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

---------

Signed-off-by: Wei Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/XL Denotes a PR that changes 500-999 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants