[Bug] Aggregation table retract issue

[zhongyujiang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

core

Minimal reproduce step

Have writen some unit tests to demonstrate this issue, see master...zhongyujiang:incubator-paimon:gh/retract-issue

What doesn't meet your expectations?

The retraction result of the agg table might be incorrect:

• the early aggregation before write will merge retract records with the same key into a single +I record
• during compaction, retract records are eliminated, even if there are still records with the same key at lower levels in the LSM tree that are supposed to be retracted

These two points will cause retraction on agg table to fail, please see the unit tests I wrote in the above link.

It seems we need to handle retracts in a similar way to how we handle delete tombstones in LSM, but that would result in both an insert and a delete record for the same key existing within a single sorted run, which might be challenging to handle I guess?

cc @JingsongLi @tsreaper could you please take a look? Thanks

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!