Reduce contention inside RocksDB during OldReceipts #6890
Conversation
benaadams
requested review from
asdacap,
MarekM25,
LukaszRozmej and
kamilchodola
There was a problem hiding this comment.
As it is in block tree it may bottleneck other use cases like json rpc - I'm not fan of this workaround, can we try changing allocator?
If not should we make this lock in Receipts Sync only?
In the past we were building rocks db so bringing it back shouldn't be too complicated. @rubo and @matilote should have knowledge about it.
| // We take the lock here instead to reduce contention on the allocator in the db. | ||
| // A better solution would be to change the allocator https://github.com/NethermindEth/nethermind/issues/6107 | ||
| using var handle = _allocatorLock.Acquire(); | ||
|
|
||
| block = _blockStore.Get( |
There was a problem hiding this comment.
Probably should be in BlockStore. There is an extension method that also does the deserialization. Probably pass the lock into the method so that the deserialization is not locked.
There was a problem hiding this comment.
Will have allocated and deallocted returning the memory to the pool to be used for next block at this level? So also use less memory and not contend on free?
|
Is this like a windows specific behaviour? |
My traces are locks inside RtlAllocateHeap which is Windows specific, so I assume the behaviour on Linux will be different.
|
|
Is this only during OldBodies/OldReceipts? What about after that? |
benaadams
changed the title
Yes, is perfectly fine and back to normal after (with or without additional lock); is more in OldReceipts than OldBodies as OldBodies is just setting and not making large allocations; whereas OldReceipts is pulling all the Bodies out of the db so using the db allocator heavily |
Which should prevent multiple concurrent json rpc requests for blocks bottlenecking block processing also? |
|
I assume oldreceipts slow down also? |
I think it was faster (on Windows); will try again. However if the allocator is actually working better in parallel (e.g. tcmalloc or Jemalloc) then it would be slower since it makes it single thread at this point. Not sure how the default Linux allocator used by RocksDb performs. |
|
On Windows Pre-change 9hrs 24mins for first 56% of receipts Post-change 56 mins for remaining 42% |
Resolves #4642
Changes
_blockStore.GetinBlockTreeto move the contention outside of the RocksDB allocator (which then heavily hits other RocksDB processing like block processing) and into the calling code (OldReceipts/OldBodies)Before
After
Types of changes
What types of changes does your code introduce?
Testing
Requires testing
Documentation
Requires explanation in Release Notes
Significantly reduces block processing times
Remarks
On Windows was seeing block processing times of up to 24seconds, this brings it down to mostly in range with the occasional spike to 2secs