fix(shwap/bitswap): Blockstore.GetSize: getting size with no compute #3894
Conversation
Wondertan
requested review from
renaynay,
vgonkivs,
walldiss and
cristaloleg
as code owners
Wondertan
force-pushed
the
shwap-bitswap-get-size
branch
2 times, most recently
from
2d98169
to
2bef3ec
Compare
Wondertan
force-pushed
the
shwap-bitswap-get-size
branch
from
2bef3ec
to
ad5a7b5
Compare
|
If GetSize is used only for prioritisation of bitswap server hanling of requests, perhaps don't really need to calculate sizes. Alternative could be to return const values depending on type of the cid and our estimate of proper priority / load of handling. For examples:
This would allow us to remove the need for heavy reads to get the priority for serving of requests and will easy the pressure on cache. |
|
In sync, we agreed to keep the Size computation based on real blocksize to be safe. Making all sizes fixed to max block size may limit bitswap's throughput for small block sizes. |
| @@ -71,6 +74,12 @@ func (sb *SampleBlock) Height() uint64 { | |||
| return sb.ID.Height | |||
| } | |||
|
|
|||
| func (sb *SampleBlock) Size(ctx context.Context, acc eds.Accessor) (int, error) { | |||
| squareSize := acc.Size(ctx) | |||
| sampleSize := libshare.ShareSize + share.AxisRootSize*int(math.Log2(float64(squareSize))) | |||
There was a problem hiding this comment.
is there no helper util we can use to determine proof size for a share based on dah?
There was a problem hiding this comment.
I am not aware of such
|
It is covered |
Bitswap prioritizes peers based on their active/pending work and the priority that peers set for requests(work) themselves. The prioritization happens on the
Getoperation ofBlockstorenotGetSize. However, we currently do the most expensive work inGetSizefollowed byGet. It is still part of the same WANT_BLOCK request(since #3813), andGetSizework is usually cached forGetto catch it. Still, the prioritization makes it so that the time betweenGetSizeandGetcan be pretty long, inducing cache misses and redoing the same work again, which is confirmed by profiles. Also, doing the expensive part inGetSizeavoids Bitswap server's rate limiting.All this brings the need to make
GetSizeas lightweight as possible, leaving actual proof computed toGet. This PR achieves this by only opening EDS and reading up its size, and then computing the expected size based on constants. TheGetSizestill does expensive file opening but avoids the precious computing to simply report the size of the data.P.S. The
RowNamespaceDataBlockstill does the expensive compute as previously, as there is no easy way to get the size of RND data besides reading it. We could only do the reading part tho without computing rn by adding a new method to Accessor, but that's not worth it ATM.🎱 mb blonks certified