Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: QueryNode's estimation of memory usage when loading a deltalog is unreasonable #36031

Closed
1 task done
XuanYang-cn opened this issue Sep 6, 2024 · 1 comment
Closed
1 task done

[Bug]: QueryNode's estimation of memory usage when loading a deltalog is unreasonable #36031

XuanYang-cn opened this issue Sep 6, 2024 · 1 comment
Assignees
@XuanYang-cn
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@XuanYang-cn
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Most of the times, deltalog's MemorySize is actually the memory size, and we're already using streaming desrialize of deltalog, so the reasonable estimation of deltalog loading memory usage should be 1.

However, there'be some segment from Milvus2.3 branch, and the MemorySize will fallback to LogSize(the compressed DiskSize of deltalog), in this case, we should also fallback to using the 50 times Factor to estimate the possible 50 times of the compression rate.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@XuanYang-cn XuanYang-cn added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 6, 2024
@XuanYang-cn XuanYang-cn self-assigned this Sep 6, 2024
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Sep 6, 2024
@XuanYang-cn
fix: Change deltalog memory estimation fator to one
2af9788 
See also: milvus-io#36031

Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn XuanYang-cn mentioned this issue Sep 6, 2024
Merged
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Sep 6, 2024
@XuanYang-cn
fix: [cp]Change deltalog memory estimation fator to one
3c0070e 
See also: milvus-io#36031
pr: milvus-io#36033

Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn XuanYang-cn mentioned this issue Sep 6, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Sep 6, 2024
@XuanYang-cn
fix: [cp]Change deltalog memory estimation factor to one (#36035)
64e109d 
See also: #36031
pr: #36033

Signed-off-by: yangxuan <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Sep 6, 2024
@XuanYang-cn
fix: Change deltalog memory estimation factor to one (#36033)
7859faf 
See also: #36031

Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn
Copy link
Contributor Author

/close

chyezh pushed a commit to chyezh/milvus that referenced this issue Sep 11, 2024
@XuanYang-cn @chyezh
fix: Change deltalog memory estimation factor to one (milvus-io#36033)
c6a1760 
See also: milvus-io#36031

Signed-off-by: yangxuan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@XuanYang-cn XuanYang-cn

Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@XuanYang-cn