Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Splitstore: support on-disk marksets using badger #6833

Merged
merged 16 commits into from
Jul 23, 2021

Conversation

vyzo
Copy link
Contributor

@vyzo vyzo commented Jul 22, 2021

Very useful if you run out of memory during compaction.

@vyzo vyzo added team/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs epic/splitstore labels Jul 22, 2021
@vyzo vyzo force-pushed the feat/splitstore-badger-markset branch 2 times, most recently from 899529d to 0793712 Compare July 23, 2021 07:19
@vyzo vyzo force-pushed the feat/splitstore-badger-markset branch from cac58e9 to 9faa3d5 Compare July 23, 2021 09:47
@vyzo
Copy link
Contributor Author

vyzo commented Jul 23, 2021

Notes:

  • marking is about twice as slow than with the in-memory map, but we have no memory pressure.
  • block validation time is unaffected, thanks to the FileIO badger options in the markset.
  • removed the markset occurs check from trackTxnRefs so as not to hit the markset; that's a non-critical optimization that does have a small effect in transient memory usage, but much better than stalling i/o access.
  • compaction time is about 50% longer than the in-memory mapset, but still under 2hrs
  • added some documentation for configuration options in the README.

@vyzo vyzo marked this pull request as ready for review July 23, 2021 09:59
@vyzo vyzo requested a review from a team as a code owner July 23, 2021 09:59
@vyzo vyzo requested a review from Stebalien July 23, 2021 09:59
@jacobheun jacobheun added this to the 1.11.1 milestone Jul 23, 2021
blockstore/splitstore/README.md Show resolved Hide resolved
blockstore/splitstore/README.md Show resolved Hide resolved
blockstore/splitstore/markset_badger.go Show resolved Hide resolved
blockstore/splitstore/markset_badger.go Show resolved Hide resolved
blockstore/splitstore/markset_badger.go Show resolved Hide resolved
blockstore/splitstore/markset_badger.go Show resolved Hide resolved
blockstore/splitstore/splitstore_compact.go Show resolved Hide resolved
blockstore/splitstore/splitstore_compact.go Show resolved Hide resolved
@magik6k magik6k enabled auto-merge July 23, 2021 17:37
@magik6k magik6k merged commit 02a37a1 into master Jul 23, 2021
@magik6k magik6k deleted the feat/splitstore-badger-markset branch July 23, 2021 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants