Enable heap profiling

[CalvinNeo]

What is changed and how it works?

Issue Number: Close #xxx

What's Changed:

1. Enable heap_activate and heap_deactivate and heap_list API
   Note that TiKV no longer uses these APIs. They have switched to a continuous heap profiling.
   We can switch later to that, after we are assured that the deadlock problem is resolved by the latest pprof-rs.
2. We no longer uses lg_prof_interval, since it can't be modified at runtime.
3. It should cowork with TiFlash: Enable and inactive prof when start tiup#2437

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Release note

[CalvinNeo]

/retest

[JaySon-Huang]

I think we should allocate another local value for storing the "old_value" get from mallctl

[CalvinNeo]

Copied from https://github.com/tikv/jemallocator/blob/e9692ae34cd2e8d888f416c26eb1664a093807d9/jemalloc-ctl/src/raw.rs#L188

See also https://github.com/tikv/tikv/blob/9fb4aaac9bfe07db066e99397b56384d7a92b3e0/components/tikv_alloc/src/jemalloc.rs#L330

[CalvinNeo]

Changed to "write" mode

[CalvinNeo]

/retest

[CalvinNeo]

/retest

[CalvinNeo]

$ curl  "http://10.2.12.81:5765/debug/pprof/heap_activate"
activate heap profile success#
$ curl  "http://10.2.12.81:5765/debug/pprof/heap_list"
/DATA/disk4/luorongzhen/tidb-data/tiflash-5761/flash/heap-CaYe7c/000001.heap		2024-07-11 15:52:31#
$ ls -halt /DATA/disk4/luorongzhen/tidb-data/tiflash-5761/flash/heap-CaYe7c/000001.heap
-rw-r--r-- 1 tidb tidb 208K Jul 11 15:52 /DATA/disk4/luorongzhen/tidb-data/tiflash-5761/flash/heap-CaYe7c/000001.heap

Log

[2024/07/11 15:51:31.142 +08:00] [INFO] [profile.rs:149] ["periodical heap profiling is started"] [thread_id=669]
[2024/07/11 15:51:48.987 +08:00] [INFO] [profile.rs:271] ["list_heap_profiles gets 0 items"] [thread_id=668]
[2024/07/11 15:52:01.131 +08:00] [INFO] [profile.rs:271] ["list_heap_profiles gets 0 items"] [thread_id=668]
[2024/07/11 15:52:18.494 +08:00] [INFO] [profile.rs:271] ["list_heap_profiles gets 0 items"] [thread_id=669]
[2024/07/11 15:52:31.161 +08:00] [INFO] [profile.rs:285] ["a heap profile is dumped to /DATA/disk4/luorongzhen/tidb-data/tiflash-5761/flash/heap-CaYe7c/000001.heap"] [thread_id=668]

[CalvinNeo]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• HACKED_OWNERS [JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-12 03:13:45.584705785 +0000 UTC m=+586522.819939898: ☑️ agreed by JinheLin.