SegmentReadTaskScheduler::add took too long.

[SeaRise]

Bug Report

[2024/05/02 14:11:37.563 +00:00] [INFO] [SegmentReadTaskScheduler.cpp:58] ["Added, pool_id=47879 table_id=7511 block_slots=48 segment_count=4 pool_count=123 cost=120437219.576us do_add_cost=7.705us"] [thread_id=929]

120s

v7.5.1

[JaySon-Huang]

The root cause is since v7.5, we may meet some DeltaMergeStore with only 0 segment in it. When the read task created, it will add some SegmentReadTaskPool without any segment task.

[2024/05/02 14:20:25.054 +00:00] [INFO] [SegmentReadTaskScheduler.cpp:58] ["Added, pool_id=49535 table_id=7514 block_slots=48 **segment_count=0** pool_count=116 **cost=154525318.043us** do_add_cost=3.077us"] [thread_id=3088]

It may make SegmentReadTaskScheduler::scheduleMergedTask return run_next_schedule_immediately=true. And the mtx always required by SegmentReadTaskScheduler::schedule, making SegmentReadTaskScheduler::add starve

tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReadTaskScheduler.cpp

Lines 182 to 191 in 9970e49

	std::optional<std::pair<uint64_t, std::vector<uint64_t>>> SegmentReadTaskScheduler::scheduleSegmentUnlock(
	const SegmentReadTaskPoolPtr & pool)
	{
	auto expected_merge_seg_count = std::min(read_pools.size(), 2); // Not accurate.
	auto itr = merging_segments.find(pool->physical_table_id);
	if (itr == merging_segments.end())
	{
	// No segment of tableId left.
	return std::nullopt;
	}

tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReadTaskScheduler.cpp

Lines 85 to 91 in 9970e49

	auto segment = scheduleSegmentUnlock(pool);
	if (!segment)
	{
	// The number of active segments reaches the limit.
	GET_METRIC(tiflash_storage_read_thread_counter, type_sche_no_segment).Increment();
	return {nullptr, true};
	}

[JaySon-Huang]

In theory, only LTS v7.5.0/v7.5.1 is affected. Because before #7437, there is at least 1 segment for empty table/partition, which won't cause this problem.