Storages: Refine memory tracker of data sharing (#8857)

[ti-chi-bot]

This is an automated cherry-pick of #8857

What problem does this PR solve?

Issue Number: close #8856

What is changed and how it works?

If has_concurrent_reader is true, we will consider the blocks to be read will being shared and set current_memory_tracker to null in

tiflash/dbms/src/Storages/DeltaMerge/File/DMFileReader.cpp

Lines 583 to 606 in fdf8bf2

	bool has_concurrent_reader = DMFileReaderPool::instance().hasConcurrentReader(*this);
	if (!getCachedPacks(column_define.id, start_pack_id, pack_count, read_rows, column))
	{
	// If there are concurrent read requests, this data is likely to be shared.
	// So the allocation and deallocation of this data may not be in the same MemoryTracker.
	// This can lead to inaccurate memory statistics of MemoryTracker.
	// To solve this problem, we use a independent global memory tracker to trace the shared column data in ColumnSharingCacheMap.
	auto mem_tracker_guard
	= has_concurrent_reader ? std::make_optional<MemoryTrackerSetter>(true, nullptr) : std::nullopt;
	auto data_type = dmfile->getColumnStat(column_define.id).type;
	auto col = data_type->createColumn();
	readFromDisk(column_define, col, start_pack_id, read_rows, skip_packs, last_read_from_cache[column_define.id]);
	column = std::move(col);
	last_read_from_cache[column_define.id] = false;
	}
	else
	{
	last_read_from_cache[column_define.id] = true;
	}
	if (has_concurrent_reader && col_data_cache != nullptr)
	{
	DMFileReaderPool::instance().set(*this, column_define.id, start_pack_id, pack_count, column);
	}

But in reality, it may not true because there are also a dt_max_sharing_column_count limitations inside ColumnSharingCacheMap. This will cause shared_column_data_mem_tracker to count data that will not actually be shared.

This PR deprecated dt_max_sharing_column_count and only use dt_max_sharing_column_bytes_for_all as the memory limitation of data-sharing.

Only need to limit the sharing memory usage in DMFileReader::readColumn.

Check List

Tests

• Unit test

• Integration test

• Manual test (add detailed scripts or steps below)

  • Run select * from lineitem concurrently.
  • The result is as the right part of the picture below
    

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[JinheLin]

/run-all-tests

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-04-12 06:03:50.691212616 +0000 UTC m=+1201492.218753157: ☑️ agreed by JinheLin.
• 2024-04-12 06:19:20.33974017 +0000 UTC m=+1202421.867280716: ☑️ agreed by JaySon-Huang.