feat(storage): add Bloom filter based event lookups #1679
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kkovaacs
force-pushed
the
krisztian/bloom-poc
branch
from
5982362
to
039318c
Compare
CHr15F0x
reviewed
Jan 17, 2024
CHr15F0x
reviewed
Jan 17, 2024
Otherwise dropping large tables is prohibitively slow (and needs too much disk space).
This change adds a new table storing Bloom filters for events emitted in that block. The filter stores the keys (prefixed with the index) and the address of the contract that emitted the event and can be used to avoid loading the receipts for blocks that do not contain matching events.
FIXME: should remove bloom when purging a block # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # On branch krisztian/events-intern-poc # Changes to be committed: # modified: Cargo.lock # modified: crates/storage/Cargo.toml # modified: crates/storage/src/bloom.rs # modified: crates/storage/src/connection/event.rs # # Changes not staged for commit: # modified: crates/crypto/src/signature/ecdsa.rs # # Untracked files: # .idea/ # TODO.md # TODO2.md # bincode-scan.txt # call-flamegraph.svg # call.json # commitment.py # crates/gateway-types/src/pending.rs # crates/rpc/fixtures/contracts/dummy_account.json # crates/stark_hash/ # crates/stark_poseidon/ # events-scan.txt # events.json # events2.json # feeder_gateway.rest # get_storage_roots.py # identity.json # json-rpc.rest # receipts-scan.txt # request.json # snapshots.txt # test.py # test1.sh # test2.sh # test3.json # test_sierra_call.sh # trace.json # trace2.json # trace3.json # trace4.json # trace_block.json #
When a filter fails to load from storage just treat it as matching.
Similar to `block_id()` but can omit a DB lookup if the block id is already a hash.
This way we avoid having to propagate the cache object around and make the fact that we use a cache transparent to the rest of the codebase.
…ansactions and receipt
When hitting the page scan limit the code now properly returns a continuation token to the client (along with the events already collected). This makes it possible for clients to eventually collect all events they are interested in by making multiple requests -- while keeping a reasonable limit on how long a single `starknet_getEvents` request takes.
kkovaacs
force-pushed
the
krisztian/bloom-poc
branch
from
b9a3480
to
ffdcc94
Compare
Mirko-von-Leipzig
approved these changes
Jan 23, 2024
Co-authored-by: Mirko von Leipzig <[email protected]>
Co-authored-by: Mirko von Leipzig <[email protected]>
Mirko-von-Leipzig
added a commit
that referenced
this pull request
Feb 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a new algorithm for serving
starknet_getEvents.New Algorithm
Instead of relying on SQLite to evaluate the filter we're now simply iterating over the block range covered by the filter, fetch events for a block and then evaluate the filter in code.
To help with performance we're adding a per-block Bloom filter that contains event addresses and keys and use that to avoid scanning event data for blocks that are known not to contain matches. Bloom filters take 2 KiB per block and are stored compressed in the database. An LRU cache helps us avoid reloading recently used Bloom filters.
The algorithm also has configurable limits on:
Upon hitting either of those limits we return with the current result set and a continuation token that can be used to continue where we've left on a subsequent request. Both of these limits (along with the Bloom filter LRU cache size) are configurable. These limits effectively allow setting an upper limit on how long a single query might take.
Storage
A Bloom filter with the parameters we've chosen takes 2 KiB per block. Compression brings that down to an average of 1430 bytes per block, so that's roughly 700 MiB for current mainnet in the database. On the other hand, this allows us to drop the
starknet_eventstable (and related indexes) which takes up slightly more than 20% of our database right now.Migration
The migration step takes considerable time. There are two major steps:
starknet_eventstable): this step takes slightly less than half an hour with mainnet on my setup.starknet_eventstable (and related stuff). Unfortunately this step required us to switch tojournal_mode=deletefor the migration, because otherwise the tables being dropped are copied to the WAL which is very expensive for the ~180 GiB of data involved."Size after" is after a vacuum (not included in migration).