executor: track the memory usage of aggregate more accurate

[wshwsh12]

What problem does this PR solve?

Issue Number: ref #33877
close #35635

Problem Summary:

1. At present, tidb's memory estimation for Golang Map only track the new buckets after expansion, and the old buckets will not be GCed immediately. Considering the worst case, the memory error will be 1.5 times. And GolangMap will definitely reach its peak value after the expansion, so we will expand the current memory statistics to 1.5 times, and reserve this part of the space for GolangMap

2. For StringSet, we shouldn't track the len(string), because map only save the pointer. We should track it when allocated.

3. The magic number in getGroupKeyMemUsage is wrong, and the unsafe.sizeof(slice) is 24.

4. We should track the memory usage of the pointer point to the PartialResult.

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  In the workload tpch-50g, try to run the aggregate function in q18: We can see that tidb tracks 39.0 GB.

[email protected]:test> desc analyze  select sum(l_quantity), l_orderkey from lineitem group by l_orderkey;
+----------------------------+--------------+-----------+-----------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+-------------------------+------+
| id                         | estRows      | actRows   | task      | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                                 | operator info                                                                                                                          | memory                  | disk |
+----------------------------+--------------+-----------+-----------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+-------------------------+------+
| HashAgg_11                 | 73908224.00  | 75000000  | root      |                | time:2m34.1s, loops:73247, partial_worker:{wall_time:49.929152551s, concurrency:5, task_num:568, tot_wait:22.124151295s, tot_exec:3m18.668402668s, tot_time:4m5.159963839s, max:49.929101236s, p95:49.929101236s}, final_worker:{wall_time:2m34.147709325s, concurrency:5, task_num:25, tot_wait:4m0.420835575s, tot_exec:8m42.60698766s, tot_time:12m43.027854689s, max:2m34.147671304s, p95:2m34.147671304s} | group by:test.lineitem.l_orderkey, funcs:sum(Column#19)->Column#18, funcs:firstrow(test.lineitem.l_orderkey)->test.lineitem.l_orderkey | 39.0 GB(Total: 39.0 GB) | N/A  |
| └─TableReader_12           | 73908224.00  | 75000433  | root      |                | time:15.1s, loops:569, cop_task: {num: 568, max: 1.25s, min: 468.5ms, avg: 616.8ms, p95: 827ms, max_proc_keys: 618925, p95_proc_keys: 528090, tot_proc: 5m25.5s, tot_wait: 410ms, rpc_num: 568, rpc_time: 5m50.3s, copr_cache: disabled}                                                                                                                                                                       | data:HashAgg_5                                                                                                                         | 90.7 MB                 | N/A  |
|   └─HashAgg_5              | 73908224.00  | 75000433  | cop[tikv] |                | tikv_task:{proc max:828ms, min:446ms, p80:595ms, p95:638ms, iters:293177, tasks:568}, scan_detail: {total_process_keys: 300005811, total_process_keys_size: 59595430182, total_keys: 300006379, rocksdb: {delete_skipped_count: 0, key_skipped_count: 300005811, block: {cache_hit_count: 977687, read_count: 0, read_byte: 0 Bytes}}}                                                                         | group by:test.lineitem.l_orderkey, funcs:sum(test.lineitem.l_quantity)->Column#19                                                      | N/A                     | N/A  |
|     └─TableFullScan_10     | 300005811.00 | 300005811 | cop[tikv] | table:lineitem | tikv_task:{proc max:696ms, min:335ms, p80:466ms, p95:503ms, iters:293177, tasks:568}                                                                                                                                                                                                                                                                                                                           | keep order:false                                                                                                                       | N/A                     | N/A  |
+----------------------------+--------------+-----------+-----------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+-------------------------+------+
4 rows in set (3 min 25.79 sec)

See the heap profile: about 40495 MB(39.5 GB) memory inuse.

See the graph

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• XuHuaiyu
• guo-shaoge

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/1f24e6fc3cc66963e3e78c169d22372cac353e43

[XuHuaiyu]

ToPtr?

[wshwsh12]

No, because type aggPartialResultMapper map[string][]aggfuncs.PartialResult

[wshwsh12]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 59d4e60

[sre-bot]

TiDB MergeCI notify

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci/integration-cdc-test	🟢 all 34 tests passed	27 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 11 tests passed	11 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	7 min 45 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	7 min 29 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	7 min 5 sec	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	6 min 56 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 14 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	4 min 4 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 54 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed