Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

feat: optimize tcmalloc release memory #343

Merged
merged 10 commits into from
Nov 29, 2019
Merged

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Nov 28, 2019

What this pr solved

Pull request #278 add a new feature that releasing memory periodically by tcmalloc ReleaseFreeMemory. ReleaseFreeMemory will release all tcmalloc reserved and not-used memory back to operating system. Releasing memory will lock page heap, the structure used to allocate memory from operating system and release memory back to operating system, shared by all threads. As a result, releasing all free memory will lock page heap for a long time and block all threads, more seriously, it will affect cluster performance. More tcmalloc information can be find in tcmalloc website.

So we optimize the releasing memory method, which is similar to kudu's way (kudu gc_memory).
We use tcmalloc statistics below:

'tcmalloc.pageheap_free_bytes' - tcmalloc reserved but not-used memory
'generic.current_allocated_bytes' - application whole allocated memory

We set a maximum percentage p, we assume that tcmalloc reserved but not-used memory should not be greater whole_used_memory * p, if exceed this threshold, we will release the exceeding memory by ReleaseToSystem. To avoid locking page heap for a long time, we only release 1MB at a time.

Besides, we make such percentage mem_release_max_reserved_mem_percentage configurable dynamically. Cluster administrator can show and change this value by remote_command.

>>> remote_command -t replica-server replica.mem-release-max-reserved-percentage 5
COMMAND: replica.mem-release-max-reserved-percentage 5

CALL [replica-server] [ip1:port1] succeed: OK
CALL [replica-server] [ip2:port2] succeed: OK
CALL [replica-server] [ip3:port3] succeed: OK

Succeed count: 3
Failed count: 0

>>> remote_command -t replica-server replica.mem-release-max-reserved-percentage
COMMAND: replica.mem-release-max-reserved-percentage

CALL [replica-server] [[ip1:port1] succeed: mem-release-max-reserved-percentage=5
CALL [replica-server] [[ip2:port2] succeed: mem-release-max-reserved-percentage=5
CALL [replica-server] [[ip3:port3] succeed: mem-release-max-reserved-percentage=5

Succeed count: 3
Failed count: 0

Config updates

Add two new config

mem_release_check_interval_ms = 3600000;
mem_release_max_reserved_mem_percentage = 10;

remove config mem_release_interval_ms

acelyc111
acelyc111 previously approved these changes Nov 29, 2019
@neverchanje
Copy link
Contributor

https://github.com/apache/kudu/blob/master/src/kudu/util/process_memory-test.cc kudu 的单测为什么不一并搬过来?

@hycdong
Copy link
Contributor Author

hycdong commented Nov 29, 2019

https://github.com/apache/kudu/blob/master/src/kudu/util/process_memory-test.cc kudu 的单测为什么不一并搬过来?

这个单测测试的kudu的函数CurrentConsumption,没有GcTcmalloc的单测

if (!dsn::buf2int32(args[0], percentage) || percentage <= 0 || percentage >= 100) {
result = std::string("ERR: invalid arguments");
} else {
_mem_release_max_reserved_mem_percentage = percentage;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个 remote command 还是建议抽出来一个函数然后写单测

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 这几处代码都很类似,可以抽象出一个函数来, 几个remote command都复用。另起一个PR做吧

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

嗯,remote command整个可以抽出来,之后的pr再做吧

@hycdong hycdong merged commit b904de6 into XiaoMi:master Nov 29, 2019
Smityz added a commit to Smityz/rdsn that referenced this pull request Nov 29, 2019
feat: optimize tcmalloc release memory (XiaoMi#343)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants