proposal: Added proposal for vertical block sharding (limit size of block) #3390
Conversation
…k size). Related to #1424 and #3068 Signed-off-by: Bartlomiej Plotka <[email protected]>
|
|
||
| * Add [`Cuckoofilter`](https://github.com/seiflotfy/cuckoofilter) to improve query performance on store GW as well with vertically sharded blocks. | ||
|
|
||
|  |
There was a problem hiding this comment.
This image is missing? https://bdupras.github.io/filter-tutorial/ link works 👍
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
Very relevant (:
Update: We are planning to propose this issue for LFX
…On Sat, 30 Jan 2021 at 23:53, stale[bot] ***@***.***> wrote:
Is this still relevant? If so, what is blocking it? Is there anything you
can do to help move it forward?
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#3390 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABVA3O3BBL76453MISUQYX3S4SLYFANCNFSM4THO4I2Q>
.
|
| type: proposal | ||
| menu: proposals | ||
| status: proposed | ||
| owner: @bwplotka |
There was a problem hiding this comment.
Feel free @Biswajitghosh98 to put yourself too
| of 2w time range compaction for Thanos this can result compaction to be not that effecting, casing block to grow to enormous sizes (1TB with index | ||
| sizes over 60 GBs) | ||
| * Recently Thanos and Prometheus enabled [vertical compaction](../components/compact.md#vertical-compactions) (under hidden flag). This introduced | ||
| abilities to do backfilling (importing blocks) and offline deduplications. |
There was a problem hiding this comment.
One more case when this is happening:
Huge Receiver rings are building potentially arbitrary size blocks for same tenants, so even simple case like:
We have 6 receive and a single tenant. We configure hashing to distribute write load to 6 instances. Let's simplify the case by saying there is no replication. This will create 6 huge blocks. Now compactor will group them into huge one.
| * There is known limitation for Prometheus index file size. It can't exceed 64GB due to [uint32 used for postings](https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/), | ||
| see [issue](https://github.com/thanos-io/thanos/issues/1424) and [code](https://github.com/prometheus/prometheus/blob/3d8826a3d42566684283a9b7f7e812e412c24407/tsdb/index/index.go#L279). | ||
| While there is work to lift this limitation and allow bigger indexes, there will be **some** other limit anyway. | ||
| * Compaction is not streaming bytes directly from & to object storage (yet). This is due to its current index format, which is [being worked on](https://github.com/thanos-io/thanos/issues/3389). |
There was a problem hiding this comment.
It is worth mentioning the current status. Chunk streaming is possible on disk and is being scheduled (not worked on yet), but index streaming is not possible currently and won't be for longer time.
|
|
||
| **We propose to vertically shard all blocks which goes beyond the specified index size limit during the compaction process.** | ||
|
|
||
| * Special metadata in meta.json indicating a shard. |
There was a problem hiding this comment.
- One problem to solve, is how to perform compaction from X sources that output in Y (where Y > 1) blocks.
- How to ensure multiple compactors can distribute compaction without strong coordination.
- How to ensure multiple store gateways can exclude certain blocks from lookup process without directly touching them. (aka Bloom/Cuckoo filter)
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
Relevant, but far in progress, unfortunately. |
|
Newer version #4191 |
Related to #1424 and #3068
Signed-off-by: Bartlomiej Plotka [email protected]