Tracking: Large dataset insert and read

[BohuTANG]

Summary

Insert:

• unexpected TableVersionMismatched error in the execution of copy into statement #7943

Read:

• feat: make table.read_partitions distributed #7805
• Feature: support more kinds of statistics information #7314
• feat: add prefix of the UUID to unlimit s3 quota #8229
• Tracking memory spill to external storage #8696
• Feature: add per segment number of blocks info to snapshot #8817
• feat: shuffle segments during distributed pruning #8793
• feat: explain show how many percent min/max index, bloom filter index prunned #8837
• feat: try to improve object storage io read #9335
• refactor(query): split fuse source to read data and deserialize #9353
• refactor(query): avoid io copy in read parquet data #9365
• refactor(query): add uncompressed buffer for parquet reader #9379
• refactor: disable parquet encoding due to performance issue #9412

Unload:

• Feature: Copy unload table support multi dest files #7618

Index:

• BUG: feed estimated number of distinct values to bloom filter #7825
• Feature: Trim Strings during the construction of min/max statistics  #7829
• Tracking issue of bloom filter refactoring #7748
• feat: cache segment block meta info to disk #7842
• feat(cluster): experimental distributed eval index #7867
• feat: parallel bloom index pruning #7893
• feat: add INT type to bloom filter #7990
• Feature: make bloom filter index cache configurable #8279
• add distinct value counting to table meta #8465

Data:

• feat: TTL for fuse table #7828
• feat: drop table atomic #7847
• bug: reqwest::Error when read a lot of files but with small open files #7899
• Streaming COPY #7889
• improvement: move table compact to pipeline builder #7924
• feat: system function to detect&remove orphaned blocks of a database #7970
• feat: COPY INTO returns more status #7730
• further compact blocks when insert/load data with many threads #8311
• performance: read_escaped_string_text when COPY TSV file #8572
• bug: COPY INTO CPU load takes a long time to rise #8574
• allow executing COPY INTO in a cluster #6395
• feat: Support ON_ERROR = CONTINUE | ABORT_STATEMENT in the CopyOptions when do COPY INTO #8642
• feat: fuse block file cache #8647

Optimize:

• feat: optimize/recluster table print status(progress) in show processlist #7875
• Feature: optimize compact performance improvement #7914
• bug: optimize table hang #7956
• bug: it occurs "entity not found" error when executing select sql from a transient table by mysql jdbc #7987
• performance: fuse_snapshot very slow #8067
• bug: Failed to execute compact during insertion #8108
• feat: optimize table xx compact limit 1000 #8241
• Feature: optimize table t compact segment only #8247
• bug: OOM during OPTIMIZE TABLE COMPACT execution #8414
• bug: Memory usage decreased gradually during OPTIMIZE TABLE PURGE and will eventually be interrupted. #8415
• bug: query slow down after optimize compact #8426

Bug:

• bug: explain is not right after enable_distributed_eval_index enabled #8049
• bug: operation interrupted during create table xx as select * from xx #8053

Metrics:

• metrics: add metrics for stats in query log #8781

[BohuTANG]

Let's continue this work at #9604 v1.0