Handle snapshot in parallel if there is only one big region in raftstore v2

[CalvinNeo]

What problem does this PR solve?

Issue Number: close #8081

Problem Summary:

In this PR, we also reorganized some codes in KVStore.cpp and Region.cpp.

What is changed and how it works?

Interface changes:

1. Add an interface fn_get_config_json to get final config on Proxy's side.
2. Add an interface fn_approx_size to get approximate size of different cf only for tablet sst reader.
3. Add an interface fn_get_split_keys to try to split a raft snapshot into splits_count parts, according to write cf. It may result in less or even zero parts if the size of the snapshot is small. The returned keys are not evenly distributed.

The feature only works when in raftstore v2 and there is few concurrent prehandling.

The whole process:

1. SSTReader calls proxy for approximate size, if it meets some threshold, we will start parallel prehandling.
2. SSTFilesToBlockInputStream compute how may splits we want to divide.
3. SSTReader ask proxy for all split keys, the whole process will fallback to single thread if proxy can't return enough keys.
4. For all count split parts, we will create another count - 1 to handle in parallel.
5. If exception or other failure happens, the sub task will throw or return the error. If one sub task fails, all other prehandling subtask will be aborted.
6. Otherwise, it will return ingest files.
7. The caller thread will handle the head(leftmost) split, and wait for all other threads join. It will aggregate outputs of all subtasks. The outputs includes ingested files(in disk) and uncommitted key-value pairs(in memory)

How to split tasks by split keys?
Given a split_key[i], a subtask who runs split_id=i will handle range in (pk(split_key[i]), pk(split_key[i+1])]. Which means it will skip all versions of pk of split_key[i], and accepts all versions of pk of split_key[i+1].

The order

region start -- head split(will also do final agg) -- split key 0 -- split 0 -- split key 1 -- split 1 -- split key n -- split n -- region end

Performance

item	master	this pr
splits	1(always)	3
lines	11095216	10474970
interval between two ddls	2min	1min20s

item	master	this pr
splits	1(always)	4
lines	-	20856771
interval between two ddls	-	1min21s

Most of the time are spent on pd scheduling and ddl. If we only count the prehandle time, then we have

parallel	cost
4
1

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  use sysbench to generate data for 300s, then add tiflash replica
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[CalvinNeo]

/hold address todo(split), merge proxy's pr, performance test

[CalvinNeo]

that is before readPrefix

[CalvinNeo]

If soft_limit is not set, then it is the only split, which is the legacy case

[JaySon-Huang]

lgtm

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-10-10 06:37:52.683059455 +0000 UTC m=+1120670.270169586: ☑️ agreed by JinheLin.
• 2023-10-10 06:42:20.073018665 +0000 UTC m=+1120937.660128808: ☑️ agreed by JaySon-Huang.

[CalvinNeo]

/unhold

[CalvinNeo]

/run-all-tests

[CalvinNeo]

/run-all-tests

[CalvinNeo]

/unhold