storage: handle IngestSST using segment split

[breezewish]

What problem does this PR solve?

Issue Number: close #5237

This is the final PR for #5237. After this one, everything should be finished.

Problem Summary:

What is changed and how it works?

Some changes are extracted:

• Merge storage: Add more checks for ingestFiles #6375 first.
• Merge storage: Accepts a snapshot for segment replaceData API #6385 first.

---

1. Add: Segment::prepareIngestData and Segment::applyIngestData.

   It checks whether the segment is empty, and will call either replaceData or ingestIntoDelta.

   • Unit tests included.

2. Add DeltaMergeStore::segmentIngestData API.

   It calls Segment::prepareIngestData and Segment::applyIngestData.

3. Change the workflow of DeltaMergeStore::ingestFiles. Changes as below:

   - 1. for clear_data=false, use `ingestIntoDelta` API. Otherwise, for clear_data=true:
     
     2. Logical-split the segment according to the range of the data to ingest
   
   - 3. Use `DeltaMergeStore::replaceData` API.
   + 3. Use `DeltaMergeStore::segmentIngestData` API.

4. Add tests for concurrent ingest data.

5. Changed existing StoreIngestTest to test for both ingestBySplit and ingestByDelta.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Workload 1: Import data using BR on IDC. When all TiFlash replicas are ready, invoke 3 full table scans with 30s interval.

Metric	This PR	master	release-6.1
CPU during BR restore	600%	1300%	304%
Disk space	40.3GB	49GB (see #5950)	40.4GB
Max disk space	41.3GB	62.6GB	46.5GB
Write amp	0.33	1.58	2.34
Peak write	136MiB/s	463MiB/s	157MiB/s
Total workload time	11min	14min	53min
TiFlash replica ready time after BR	0min	0min	41min
Query time 0	16.22	17.07	24.65
Query time 1	13.28	15.70	23.47
Query time 2	12.28	14.46	23.95

Workload 2: Import CHBenchmark (warehouse=1000) data using BR on AWS EC2. When all TiFlash replicas are ready, invoke 3 full table scans with 30s interval.

Metric	This PR	master	release-6.1
CPU during BR restore	126%	288%	204%
Disk space	53.1GB	53.4GB	53.3GB
Max disk space	53.1GB	67.2GB	61.6GB
Write amp	0.68	4.51	6.91
Peak write	41MiB/s	253MiB/s	249MiB/s
Total workload time	34min	37min	59min
TiFlash replica ready time after BR	0min	0min	20min
Query time 0	1 min 5.96 sec	1 min 5.98 sec	1 min 6.49 sec
Query time 1	1 min 5.94 sec	1 min 5.96 sec	1 min 4.97 sec
Query time 2	1 min 5.93 sec	1 min 5.98 sec	1 min 6.48 sec

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Reduce resource usage when import using BR / Lightning

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• JaySon-Huang
• flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[breezewish]

Note: This recursive mutex happens when:

1. DeltaMergeStore::segmentIngestData grabs a segment update lock
2. DeltaMergeStore::segmentIngestData discovered the segment is not empty and then ingest files into the delta.
   • DeltaValueSpace::ingestColumnFiles grabs the same lock to write to delta VS.

The real problem is we do not distinguish the segment update lock with the delta lock. When both lock is needed, they are conflicted. Also, I did not changed the ingestColumnFiles API to allow passing an external lock, because other APIs in the DeltaVS do not have this pattern and I think it may not be a good idea.

[JaySon-Huang]

Usually, in DeltaMergeStore::segmentApplyXXX we will acquire the segment update lock and won't change the DeltaVS inside the "apply".
segmentIngestData brings a special pattern that may change the DeltaVS inside its "apply". But I didn't figure out a better solution

[JaySon-Huang]

Rest lgtm

[breezewish]

/run-all-tests

[JaySon-Huang]

lgtm

[breezewish]

/run-all-tests

[flowbehappy]

/merge

[ti-chi-bot]

@flowbehappy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 742f389