ethdb: improve batch size calculation #23790
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Wouldn't this change also change how often we commit batches? By halving the amount of data in a batch thus doubling how often we call |
I wouldn't put too much value on that. The idealbatchsize is an old magic constant, and I see it as more of an indicator than an scientific exact optimal value. I think we should do the change in this PR, and yes, at some point investigate if our batch sizes seem appropriate or not. |
sidhujag
pushed a commit
to syscoin/go-ethereum
that referenced
this pull request
Oct 26, 2021
This PR also counts the size of the key when calculating the size of a db batch
This was referenced Sep 23, 2022
yongjun925
pushed a commit
to DODOEX/go-ethereum
that referenced
this pull request
Dec 3, 2022
This PR also counts the size of the key when calculating the size of a db batch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the batch size calculation a bit.
In Geth there is a function called
ValueSizeof db batch, the number returned can act as an indicator to prevent accumulate too much data in a single batch.However it only counts the value size, instead of counting both key and value size. It's OK in most of cases but may lead to critical issue in some special scenarios, e.g. there are only
Batch.Put(key, nil)operations queued up in the batch, then they will all be ignored.So in this PR, key size is also counted. It won't have a big impact to the system, but try to avoid some unexpected error.