Limit size of blocks to X bytes on compaction.

[bwplotka]

This is to ensure the scalability of the compactor.

Tricky part: How to shard block, what label to put, or maybe vertical compaction?

[bwplotka]

On it now, proposal to be created.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Still in progres. Vertical block sharding has to be designed and implemented.

[bwplotka]

Proposal is half done: #3390

The plan is to team up with LFX mentee for knowledge sharing 🤗

[someshkoli]

Hey @bwplotka, I'd like to pick this up again and work on it.

Proposal is half done: #3390

From what I understand, we need to shard the block after certain size continuously so that it won't hit the size limit (64gb as for now). Also how are we planning to shard these blocks, I remember planning of breaking blocks into half of the compacted size until its below the threshold value, Any pointers on that ?

[bwplotka]

cc @Biswajitghosh98 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[oronsh]

How about a new flag to set max time range to apply compaction on? So if for example the compactor detects metadata with now - max_time > max_compaction_timerange exclude the block.
This will probably help reducing size so one could possibly set the compactor to compact only 4 hours old blocks max for example.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[alvinlin123]

Where is the proposal for this feature?

[bwplotka]

Here: #4191

[bwplotka]

Update: Currently compactor will not create bigger blocks than with the index configured with compact.block-max-index-size (hidden flag). Thus this issue can be closed.

I will add another one with more dynamic solution that actually can split blocks vertically.

[bwplotka]

Created for tracking #5437

[yeya24]

This feature is mainly about limiting block index size? Chunk sizes are not included

[BouchaaraAdil]

@bwplotka how can we control Block sizes?
weh have blocks that are over 100GB

Start Time: June 12, 2024 1:00 AM
End Time: June 18, 2024 1:00 AM
Duration: 6 days
Series: 56716960
Samples: 52166082084
Chunks: 462569560
----------------
Total size: 109.90 GiB
Chunks: 98.72 GiB (89.82%)
Index: 11.18 GiB (10.18%)
Daily: 18.32 GiB / day
----------------
Resolution: 0
Level: 4
Source: compactor

we see this error in these cases:

 | ts=2024-09-14T17:29:55.251619424Z caller=compact.go:546 level=error msg="retriable error" err="first pass of 

downsampling failed: 2 errors: downsampling to 5 min: download block 01J5WJ3G8FZ4CXAHWPVDYG4YB6: get file 

01J5WJ3G8FZ4CXAHWPVDYG4YB6/chunks/000020: The difference between the request time and the current time is 

too large.; downsampling to 5 min: download block 01J3DQ5Q98458FHBA84XG996GAA6: get file 

01J3DQ5Q16A00BA8484HVH46/chunks/000116: The difference between the request time and the current time is too large." |