Memory tracker is not accurate if read pool is enabled.

[windtalker]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. setup a TiDB cluster with 1 TiFlash node
2. Load TPCH 10 data
3. using the following script to run sql

for num in {1..concurrency}
do  
        mysql --host host_ip --port port -uroot < /tmp/test.sql &
done

wait;

The test.sql is

set tidb_enforce_mpp=1;
use tpch10;
select * from lineitem where l_shipmode like '%ab%';
select * from lineitem where l_shipmode like '%ab%';
select * from lineitem where l_shipmode like '%ab%';
select * from lineitem where l_shipmode like '%ab%';
select * from lineitem where l_shipmode like '%ab%';

With read pool disabled, the concurrency is 2, the memory usage reported by memory tracker are

[2023/09/21 09:11:23.008 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.04 GiB."] [source=MemoryTracker] [thread_id=243]
[2023/09/21 09:11:23.010 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=242]
[2023/09/21 09:11:23.010 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 2.09 GiB."] [source=MemoryTracker] [thread_id=242]
[2023/09/21 09:11:25.443 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=248]
[2023/09/21 09:11:25.444 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.02 GiB."] [source=MemoryTracker] [thread_id=250]
[2023/09/21 09:11:25.444 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 2.02 GiB."] [source=MemoryTracker] [thread_id=250]
[2023/09/21 09:11:27.851 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=254]
[2023/09/21 09:11:27.892 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.04 GiB."] [source=MemoryTracker] [thread_id=256]
[2023/09/21 09:11:30.229 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.09 GiB."] [source=MemoryTracker] [thread_id=259]
[2023/09/21 09:11:30.265 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.07 GiB."] [source=MemoryTracker] [thread_id=262]
[2023/09/21 09:11:32.590 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.06 GiB."] [source=MemoryTracker] [thread_id=265]
[2023/09/21 09:11:32.635 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.04 GiB."] [source=MemoryTracker] [thread_id=268]
[2023/09/21 09:11:32.635 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 2.04 GiB."] [source=MemoryTracker] [thread_id=268]

With read pool disabled, the concurrency is 1, the memory usage reported by memory tracker are

[2023/09/21 09:10:46.260 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.06 GiB."] [source=MemoryTracker] [thread_id=223]
[2023/09/21 09:10:46.260 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 1.06 GiB."] [source=MemoryTracker] [thread_id=223]
[2023/09/21 09:10:47.501 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.08 GiB."] [source=MemoryTracker] [thread_id=226]
[2023/09/21 09:10:47.501 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 1.08 GiB."] [source=MemoryTracker] [thread_id=226]
[2023/09/21 09:10:48.741 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.06 GiB."] [source=MemoryTracker] [thread_id=229]
[2023/09/21 09:10:48.742 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 1.06 GiB."] [source=MemoryTracker] [thread_id=229]
[2023/09/21 09:10:50.000 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=232]
[2023/09/21 09:10:50.000 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 1.05 GiB."] [source=MemoryTracker] [thread_id=232]
[2023/09/21 09:10:51.240 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=235]
[2023/09/21 09:10:51.240 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 1.05 GiB."] [source=MemoryTracker] [thread_id=235]

With read pool enabled, the concurrency is 2, the memory usage reported by memory tracker are

[2023/09/21 09:08:37.087 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 10.48 GiB."] [source=MemoryTracker] [thread_id=269]
[2023/09/21 09:08:37.090 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 288.86 MiB."] [source=MemoryTracker] [thread_id=270]
[2023/09/21 09:08:37.090 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 988.18 MiB."] [source=MemoryTracker] [thread_id=270]
[2023/09/21 09:08:38.407 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 8.28 GiB."] [source=MemoryTracker] [thread_id=275]
[2023/09/21 09:08:38.824 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 244.48 MiB."] [source=MemoryTracker] [thread_id=277]
[2023/09/21 09:08:39.801 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 3.79 GiB."] [source=MemoryTracker] [thread_id=280]
[2023/09/21 09:08:40.381 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 288.98 MiB."] [source=MemoryTracker] [thread_id=283]
[2023/09/21 09:08:41.408 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 2.69 GiB."] [source=MemoryTracker] [thread_id=286]
[2023/09/21 09:08:42.029 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 397.90 MiB."] [source=MemoryTracker] [thread_id=289]
[2023/09/21 09:08:43.009 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 4.41 GiB."] [source=MemoryTracker] [thread_id=292]
[2023/09/21 09:08:43.488 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 435.62 MiB."] [source=MemoryTracker] [thread_id=295]
[2023/09/21 09:08:43.488 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 8.89 GiB."] [source=MemoryTracker] [thread_id=295]

With read pool disabled, the concurrency is 1, the memory usage reported by memory tracker are

[2023/09/21 09:07:38.645 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 698.27 MiB."] [source=MemoryTracker] [thread_id=252]
[2023/09/21 09:07:38.645 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 698.27 MiB."] [source=MemoryTracker] [thread_id=252]
[2023/09/21 09:07:39.848 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 680.21 MiB."] [source=MemoryTracker] [thread_id=255]
[2023/09/21 09:07:39.848 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 680.21 MiB."] [source=MemoryTracker] [thread_id=255]
[2023/09/21 09:07:41.041 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 685.35 MiB."] [source=MemoryTracker] [thread_id=258]
[2023/09/21 09:07:41.041 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 685.35 MiB."] [source=MemoryTracker] [thread_id=258]
[2023/09/21 09:07:42.242 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 690.88 MiB."] [source=MemoryTracker] [thread_id=261]
[2023/09/21 09:07:42.242 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 690.88 MiB."] [source=MemoryTracker] [thread_id=261]
[2023/09/21 09:07:43.450 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 701.03 MiB."] [source=MemoryTracker] [thread_id=264]
[2023/09/21 09:07:43.450 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 701.03 MiB."] [source=MemoryTracker] [thread_id=264]

As we can see, when read pool enabled, when running queries with concurrency(even the concurrent is only 2), the memory usage reported by memory tracker is very inaccurate, for both query level and global level.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

[SeaRise]

/severity major