Support online compaction for paimon

[shidayang]

Currently, there are two main forms of compaction for Paimon. One is inline compaction, which is embedded in the write task. The other is dedicated compaction, which starts a separate Flink task to perform compaction for a specific table. If a table has multiple writes, such as partial updates, then dedicated compaction is required.

1. It is cumbersome to start a Flink job every time a table is created.
2. As the number of compaction jobs increases, the management cost becomes high. Which task merges which table? What is the status of the task? Can the merging progress keep up?
3. If compaction runs continuously, it will waste resources for tables with high peak traffic but low normal traffic. How to run compaction tasks only when a table needs to be compacted?

Therefore, initiating a discussion on online compaction means discovering tables that need to be compacted through AMS and scheduling them to a shared resource for compaction merging. Paimon users are welcome to provide feedback on whether this is necessary.

[ChenShuai1981]

In the scenario of sparse table, since the inline compaction will impact the performance of writing data, we prefer to use the dedicated compaction. So we need a companion job for each sparse paimon table. And the need of management those companion job is increased. We want to know the running status of dedicated compaction job, including the performance of compaction, how many small files be compacted to large files and how long it takes, can it catch up with the speed of writing? etc. Hope Amoro can support online compaction for paimon, just like it supports iceberg now.

[HuangFru]

If AMS supports online compaction of Paimon tables, we should also consider the following issues:

1. Should AMS manage the triggering of compaction? Or use a listener-like mechanism to obtain the trigger signal to complete the trigger compaction.

2. Is there some current limiting mechanism when multiple tables need to perform compaction tasks at the same time during peak periods? This is because there are generally not too many shared resources, otherwise it will not save resources.

[ChenShuai1981]

I think AMS should manage the compaction of all paimon tables.
Not all paimon tables suited to perform compaction tasks with shared resources. Just like the session mode and application mode execution in apache flink. Application mode suites for those large jobs per job while session mode suites for others in a whole.