Milestone of Partition Table in Sprint3 (2021/4/19 ~ 2021/6/18)

[qw4990]

Development Task

Development plan:

Brief introduction: improve the quality of all experimental features of partition tables, including direct reading, global statistics, list, and column list partition.

All test cases are in the GA matrix document.

Work items:

• Compatibility of global-stats and stats-ver1 @qw4990
  • Support FMSketch in TiKV
  • Support FMSketch in MockStore
  • Support using FMSketch to calculate NDV of indexes
  • Add more test cases
• Compatibility of global-stats and fast-analyze @rebelice
  • Make global-stats support fast-analyze executor, statistics: remove NULL value from column histogram created by fast analyze #24183
  • Add more unit test cases about global-stats and fast-analyze
• Accuracy test of global-stats @rebelice
  • Finish the accuracy test of global-stats and generate a report, the report doc
  • Adjust our global-stats merging algorithm according to the test result statistics: fix three bugs about MergePartTopN2GlobalTopN #24370, executor, statistics: remove NULL value from column histogram created by fast analyze #24183, statistics: dump FMSketch to KV only for partition table with dynamic prune mode #24453
• Direct reading @qw4990
  • Support reading partition table for index lookup reader Support reading partition table for index lookup reader #24154, execution: make IndexLookUpReader support reading partition table directly #24197
  • Support reading partition table for index merge reader Support reading partition table for index merge reader #24160, executor: make IndexMergeReader support reading partition table directly #24239
  • Make IndexReaders in the inner side of IndexJoins support direct reading Make IndexReaders in the inner side of IndexJoins support direct reading #24255, executor: make IndexReaders in the inner side of IndexJoins support direct reading #24260
  • Make IndexLookUp in the inner side of IndexJoins support direct reading Make IndexLookups in the inner side of IndexJoins support direct reading #24256, executor: make IndexLookUps in the inner side of IndexJoins support direct reading #24261
  • Use TableReader directly when accessing TiFlash partition table instead of using PartitionTableExec Use TableReader directly when accessing TiFlash partition table instead of using PartitionTableExec #24343 executor: Use TableReader directly when accessing TiFlash partition table instead of using PartitionTableExec #24344
  • Make UnionScan support direct reading and remove all code about PartitionTableExec remove all code about PartitionTableExec #24393 executor: solve the compatibility problem between UnionScan and partition tables and remove all code about PartitionTableExec #24394
• Enhancement:
  • Support PointGet for range/list partition dynamic partition pruning mode cannot use PointGet and BatchGet #24476 @zhuo-zhi
  • Support BatchGet for range/list partition
• Correctness testing 1: test dynamic pruning and partition related expression calculation @mjonss
  • Design cases on hash partitions
  • Design cases on range partitions
  • Design cases on list partitions
  • Design cases of operations on partition boundaries
  • Implement cases on hash partitions
  • Implement cases on range partitions
  • Implement cases on list partitions
  • Implement cases of operations on partition boundaries
• Correctness testing 2: test direct reading with other operators @zhuo-zhi
  • Design cases about BatchGet and PointGet
  • Design cases about Order + Limit
  • Design cases about all kinds of Joins, especially IndexLookUpJoin
  • Design cases about all kinds of Aggregations
  • Implement cases about BatchGet and PointGet for hash partition
  • Implement cases about BatchGet and PointGet for range partition
  • Implement cases about Order + Limit
  • Implement cases about all kinds of Joins, especially IndexLookUpJoin
  • Implement cases about all kinds of Aggregations
• Correctness testing 3: test direct reading with other features @qw4990
  • Design cases about Plan Cache
  • Design cases about Global Statistics
  • Design cases about Global Index
  • Design cases about SQL Bind
  • Design cases about View
  • Design cases about New Collation (Unicode CI and General CI)
  • Design cases about Clustered Index
  • Design cases about DDL Statements, like alter table schema, drop partitions
  • Implement cases about Plan Cache
  • Implement cases about Global Statistics
  • Implement cases about Global Index
  • Implement cases about SQL Binding
  • Implement cases about View
  • Implement cases about New Collation (Unicode CI and General CI)
  • Implement cases about Clustered Index
  • Implement cases about DDL Statements, like alter table schema, drop partitions
• Correctness testing 4: Randgen testing @rebelice
  • Modify clustered-index-rand-gen to make it support generating queries on partition tables, the design doc
  • Add some cases into clustered-index-rand-gen to generate testing queries (an example)
  • Integrate clustered-index-rand-gen into UTF to run our test cases
• Performance testing:
  • Design doc, IN REVIEW
  • Finish our perf-test and generate a report
• Scenario testing:
  • Design doc, IN REVIEW
  • Finish all scenario tests and generate a report
• Bugfix:
  • read_from_storage with new partition impl cannot works #20372
  • histogram buckets has NULL value crated by fast analyze #24171
  • panic when reading a partition table by IndexLookUpJoin with IndexReader #24259
  • panic when reading a partition table by IndexLookUpJoin with IndexLookUpReader #24307

[zz-jason]

we also need to fix the existing bugs, like #20372

[qw4990]

we also need to fix the existing bugs, like #20372

OK, I'll go through all existed issues about the partition table add them here later.

[zz-jason]

1. for the milestones, another key milestone is to disable static pruning and enable dynamic pruning by default.
2. for tests, I wish to add more compatibility tests:
   • upgrade tidb cluster from 4.0 to the current release, check whether the dynamic pruning works
   • downgrade tidb cluster from the current release to 4.0, check whether the static pruning works

[rebelice]

There is bug #24171 here that prevents fast analyze from generating global stats.
(I can't modify the description, so I just comment here

[zz-jason]

#22408 mentioned that there might be performance and resource issues in backup scenario, please take a look

[zz-jason]

we need to add compatibility tests with other components in the cluster, for example: TiFlash MPP, BR, Dumpling, Lightning, Binlog, TiCDC

[tisonkun]

Hi @qw4990 ! Is the design of partition table persisted as this file? Or anything should be updated?

[qw4990]

Hi @qw4990 ! Is the design of partition table persisted as this file? Or anything should be updated?

Actually there are many updates after the proposal you mentioned. Here is a simple doc that may help you track the current progress. I think we will sort out and update some documents about partition table after finishing this milestone.

[tisonkun]

Hi @qw4990 ! Is the design of partition table persisted as this file? Or anything should be updated?

Actually there are many updates after the proposal you mentioned. Here is a simple doc that may help you track the current progress. I think we will sort out and update some documents about partition table after finishing this milestone.

Thanks! I'll try to take a look when I stage my current works.

FYI when I refer to this project in our developer discussion forum(https://internals.tidb.io/t/topic/97/2), I said

It is to final comment period before we make the consensus, and thus we do not trace back to its design document. But both user document and dev document for partition table is required I think. @qw4990

Please confirm whether it is precise and I'll appreciate it if you can reply on the forum topic.