Tracking issue for compaction offloading

[LeslieKid]

Describe This Problem

We found in production that the speed of sst compaction is unable to keep up with the speed of sst generation, leading to poor query performance. However we are unable give more resource to compaction to solve the problem because query/write is more important than compaction in the same node.

It is really hard to do a trade-off about resource allocation among query, write and compaction in lsm model. We want to compact the generated small ssts as fast as possible, but we can't tolerate its influence to query/write. And finally I think offload the compaction to the seperated nodes may be the key for it.

Proposal

The following is the architecture for compaction offloading.

To support compaction offloading, we need:

• Horaedb node supports submitting the compaction task to remote feat: support horaedb submit compaction task to remote #1563
  • Introduce runnable compaction node and expose the api for horaedb node to ask for remote compaction service (Execute Plane)
  • Impl remote mode compactor supporting compaction offload
• Horaemeta supports managing the compaction nodes (Control Plane)
  • Impl the ability to monitor the compaction servers (Monitor)
  • Impl scheduling algorithm for load balance and expose the api for horaedb node to get the proper remote compaction node (Scheduler)
• Integration tests for compaction test: add integration test for compaction offload #1573
  • See Integration test for compaction offload #1571 for more details

Additional Context

This issue replaces issue #1480. Please close issue #1480 as it is outdated.

incubator-horaedb-proto#133 is highly related to this issue.

[Rachelint]

looks great!

[Rachelint]

I think in the first stage, we can treat the compaction cluster as same as the horaedb cluster.
And we just reuse the ability about monitoring and scheduling for horaedb cluster in horaemeta.

And we can start to desigin the specific schedule strategy for compaction cluster after it can work.

[LeslieKid]

I think in the first stage, we can treat the compaction cluster as same as the horaedb cluster. And we just reuse the ability about monitoring and scheduling for horaedb cluster in horaemeta.

And we can start to desigin the specific schedule strategy for compaction cluster after it can work.

Thanks for your suggestion! I think it's a good idea to first implement a simple working version.

[Rachelint]

I think Horaedb node supports submitting the compaction task to remote may be similar as Compaction node supporting remote compaction service .

I guess the third step can be integration test, test is very important actually.

[Rachelint]

Maybe you will be insterested when writing test
https://github.com/apache/horaedb/tree/main/integration_tests/dist_query