[fix] : free the memory of stmt holded by ComputationWrapper manually #13646
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jensenojs
force-pushed
the
fix_moc2035
branch
2 times, most recently
from
56393e9 to
2f40e71
Compare
matrix-meow
added
size/S
jensenojs
force-pushed
the
fix_moc2035
branch
3 times, most recently
from
9a8a4c1 to
1cf0256
Compare
对比下图对比了main和修改之后的分支在导入slooong_003同一段时间之后的内存pprof情况. main
free掉AST之后
可以看到, 存着的ast本身的内存都比free掉这部分内存之后的mo占据的所有的内存都大了... 首先这个pr的修改并不理想, 但是我们没有合适的工具找到是什么东西hold住了cwft导致这部分内存没有被gc, 后面建个issue跟踪这个事情. 等viewcore修好之后这类问题可能会好定位一些, golang/go#57447 thanks for 岳声哥 @reusee |
jensenojs
force-pushed
the
fix_moc2035
branch
from
09bfe48 to
5f61411
Compare
7 tasks
jensenojs
changed the title
daviszhen
approved these changes
Dec 26, 2023
mergify bot
pushed a commit
that referenced
this pull request
Dec 26, 2023
cherry-pick from : #13646 Approved by: @daviszhen, @qingxinhome, @sukki37
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Which issue(s) this PR fixes:
issue # https://github.com/matrixorigin/MO-Cloud/issues/2035
What this PR does / why we need it:
在导入slooong_003的时候发现内存上涨的速度非常夸张而且没有free掉, pprof之后发现是stmts导致的,
在32GB, 16core的机器上用单CN, insert的方式导入slooong_003
手动置为nil之后在导入slooong_003时的最大内存占用在18GB, 随后回落到16GB
作为对比main在开始导入数据一分钟左右占用的内存就会突破16GB, 基本稳定在31GB左右, 有不小的概率oom