Alpha: Enable bloom filter caching #5552
Conversation
Badger supports caching of SST bloom filters, this PR enables caching bloom filters in ristretto.
jarifibrahim
requested review from
manishrjain and
vvbalaji-dgraph
as code owners
There was a problem hiding this comment.
Small comment.
Reviewed 3 of 3 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @jarifibrahim, @manishrjain, and @vvbalaji-dgraph)
worker/server_state.go, line 157 at r1 (raw file):
WithKeepBlockIndicesInCache(true). WithKeepBlocksInCache(true). WithMaxBfCacheSize(500 << 20) // 500 MB of bloom filter cache.
Any reasoning/benchmarking around having default of 500 MB would be nice. Assuming max size of bloom filter as 1MB for each table. Cache can have bloom filtes for 500 SSTs.
There was a problem hiding this comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @manishrjain, and @vvbalaji-dgraph)
worker/server_state.go, line 157 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
Any reasoning/benchmarking around having default of 500 MB would be nice. Assuming max size of bloom filter as 1MB for each table. Cache can have bloom filtes for 500 SSTs.
No. This is just an estimated size which I think should be enough to store a decent amount of bloom filters.. I did not do any benchmarks.
There was a problem hiding this comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @manishrjain, and @vvbalaji-dgraph)
Badger supports the caching of SST bloom filters. This PR enables caching bloom filters in ristretto. The bloom filter cache is enabled only for the p directory. The bloom filters for the w directory will be kept in memory. We can add caching for it later if necessary. This PR also removes old stale todos. (cherry picked from commit 92328a7)
Badger supports the caching of SST bloom filters. This PR enables caching bloom filters in ristretto. The bloom filter cache is enabled only for the p directory. The bloom filters for the w directory will be kept in memory. We can add caching for it later if necessary. This PR also removes old stale todos. (cherry picked from commit 92328a7)
Badger supports the caching of SST bloom filters. This PR enables caching bloom filters in ristretto. The bloom filter cache is enabled only for the p directory. The bloom filters for the w directory will be kept in memory. We can add caching for it later if necessary. This PR also removes old stale todos. (cherry picked from commit 92328a7)
Badger supports the caching of SST bloom filters. This PR enables caching bloom filters in ristretto. The bloom filter cache is enabled only for the p directory. The bloom filters for the w directory will be kept in memory. We can add caching for it later if necessary. This PR also removes old stale todos. (cherry picked from commit 92328a7)
Badger supports the caching of SST bloom filters. This PR enables caching bloom filters in ristretto. The bloom filter cache is enabled only for the p directory. The bloom filters for the w directory will be kept in memory. We can add caching for it later if necessary. This PR also removes old stale todos.
Badger supports caching of SST bloom filters, this PR enables caching
bloom filters in ristretto.
The bloom filter cache is enabled only for the
pdirectory. The bloom filters for thewdirectory will be kept in memory. We can add those later if necessary.This PR also removes old state todos.
This change is