Reclaim ColumnFileBig that cannot be contained by any segment

[breezewish]

Enhancement

Suppose we write a ColumnFileBig into the memtable of a segment without any delta:

-Inf                            +Inf
  |<------------------------------>| Segmemt
                     |<-CFBig->|

Then physical split happens:

-Inf                            +Inf
  |<--------------->|<------------>| Segmemt
                     |<-CFBig->|

The ColumnFileBig is now referenced by both two segments. And,

1. The right segment may trigger a delta merge because the delta layer is big.

2. The left segment will not trigger a delta merge because the delta is still empty -- Its referenced CFBig is not contained in the segment.

As a result, the ColumnFileBig is kept being referenced and not recycled, until user manually triggers a DeltaMerge for all segments.

This happens when we ingest SSTs quickly (using a higher ingest concurrency), result in 25% space amplification in my experiment.

[flowbehappy]

I wonder why "Its referenced CFBig is not contained in the segment."?

[breezewish]

I wonder why "Its referenced CFBig is not contained in the segment."?

We intentionally made a filter when calculating the delta rows and delta bytes, to filter out packs that is not contained by the segment, to be more precise:

tiflash/dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFileBig.cpp

Line 33 in 014e55c

void ColumnFileBig::calculateStat(const DMContext & context)

So in the example scenario, we will have delta (logical) rows|bytes = 0 (which is, reasonable).

[flowbehappy]

I get it. Although a CFBig is contained by a segment, the data in the CFBig is not. Because all of the data in CFBig is filtered out by segment's range.