Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: enable rocksdb prefix seek mode to improve hashkey scan. #421

Closed
neverchanje opened this issue Nov 14, 2019 · 0 comments
Closed

perf: enable rocksdb prefix seek mode to improve hashkey scan. #421

neverchanje opened this issue Nov 14, 2019 · 0 comments
Labels
type/enhancement Indicates new feature requests

Comments

@neverchanje
Copy link
Contributor

neverchanje commented Nov 14, 2019

Currently, our storage engine uses rocksdb's default options to construct a bloom filter, which only checks the existence of "hashkey+sortkey". For the situation where users scan over the range of hashkey (multigetrange, hashscan), the bloom filter makes no optimization if the hashkey does not exist. And rocksdb will search the corresponding SST data block, until it confirms the inexistence.

So what can we do with it? RocksDB provides an option called prefix seek mode. Its major improvement is adding prefix (the hashkey) as well as the whole key to the bloom filter. (https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#prefix-vs-whole-key)

For point lookups (multiget/get), only the whole key blooms are used. For range queries (multigetrange, hashscan, unorderedscan+hashkey prefix filtering), rocksdb responds without searching SST after the hashkey misses in the bloom filter.

One other improvement is to use hash-based memtable. Since Pegasus doesn't support cross-partition ordering, this implementation can be taken into consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

2 participants