Coroutine leaking after one TiKV got stuck

[5kbpers]

Bug Report

Affected versions

v5.1, v5.2, master

Description

For Advancing the Resolved TS, we need to verify the leadership of Raft Leaders by calling a check_leader RPC every 1s to all the other TiKVs (Here we did not filter out learners so there will call the RPC to TiFlash).
However, here we use futures::join_all to wait for all responses to get received. Then once one TiKV got stuck and the timeout is too long, some coroutines would be accumulated and probably lead to OOM.

[BusyJay]

How do you find the problem? Can pending futures also be added to memory trace?

Can yatp support memory trace? /cc @sticnarf

[5kbpers]

How do you find the problem? Can pending futures also be added to memory trace?

Can yatp support memory trace? /cc @sticnarf

Through the auto-generated jemalloc profile (start TiKV with MALLOC_CONF=prof:true,prof_active:true,lg_prof_interval:33), compared the adjacent two profiles and found the problem.

jeprof.25133.25.i25.heap.svg.zip
jeprof.25133.24.i24.heap.svg.zip

[BusyJay]

Does memory profile from dashboard work? Why it can't capture the problem before?

[hicqu]

Does memory profile from dashboard work? Why it can't capture the problem before?

Memory profile from dashboard or wget is not strong enough to help this case. I'm working on improving it now.

[NingLin-P]

The cdc component also adopt the CheckLeader mechanism and the similar usage of futures::join_all, so this issue may also happen to cluster that deployed TiCDC.