Streaming COPY

[Xuanwo]

Background

Databend will list all files before copying data:

https://github.com/datafuselabs/databend/blob/46da962d5637b35555e8f3f49885d1b58211ab41/src/query/service/src/interpreters/interpreter_copy_v2.rs#L399-L460

This will lead to:

• bug: COPY INTO command cannot import data from s3 with massive files or directories #7862

We need to introduce streaming copy which will copy simultaneously during the listing.

Problems

• GetTableCopiedFileReply will return all files
• UpsertTableCopiedFileReq requires update all copied files at once

We need to change them to streaming-friendly. For example, only check a batch of files.

Tasks

• bug: optimize get/upsert copied file info #7892
• Copy file streamingly

[Xuanwo]

Cc @lichuang

[BohuTANG]

Before this work, I think we should finish CopyInterpreterV2.list_files can be same with InterpreterCommon.list_files

[ClSlaid]

further update on this?

[Xuanwo]

Wait for #7892

[lichuang]

Wait for #7892

#7892 will be processed after finishing share table impl

[lichuang]

#7892 has been closed.

[Xuanwo]

There are other works to finish on copy side. Let's reopen it.

[BohuTANG]

I will take this, we need to consider more situations:
If COPY will be distributed, streaming reading files in StageTable:read_partitions is hard.
For now, it looks better to follow the current mechanism for copying. If a bucket has many files, now slow(list files) is by design.

cc @Xuanwo

[Xuanwo]

streaming reading files in StageTable:read_partitions is hard.

Partitions scanning and data reading are different kinds of workloads that can be executed at the same time. Adding this feature will also improve the performance of FuseTable.

With this feature, we can do copying at the same time of listing:

Given a bucket with 1000_0000 (1kw) files, every listing request will return 200 objects, and the typical list latency is 100ms. We will need 1000_0000 / 200 / 0.1 = 1.3 hour to fully scan it.

[BohuTANG]

New COPY design will be use StageTable as table engine and re-use the planner& pipeline distribution:
#8635