Das batch store chunking #2341
Merged
Das batch store chunking #2341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These are separate concerns, but were mixed together in one class.
This adds support for the batch poster in AnyTrust configuration to send batches to the DA committee in chunks of a configurable maximum HTTP POST body size. It adds new RPC methods to the daserver executable's RPC server, das_startChunkedStore, das_sendChunk, das_commitChunkedStore, which the clients created by the batch poster will automatically use if they are detected to be available on the committee server, otherwise it will fall back to the legacy das_store method. This allows an easy roll out of either the client or server first. The payloads of the new RPC methods are all signed by the batch poster. As basic DoS prevention, at most 10 uncommitted stores can be outstanding, uncommitted stores expire after a minute, and a das_startChunkedStore with the same arguments is not replayable after a minute. The batch poster should only be trying to store one batch at a time, so this should be sufficient. The new option --node.data-availability.rpc-aggregator.max-store-chunk-body-size is expressed in terms of the HTTP POST body size that the operator wants the chunk requests to stay under. 512B of padding is also added to whatever the user operator specifies here, since some proxies or endpoints may additionally count headers. This is orthogonal to settings like --node.batch-poster.max-size which control the maximum uncompressed batch size assembled by the batch poster. This should allow the batch poster to create very large batches which are broken up into small chunks to be sent to the committee servers. Once the client has received confirmation to its das_startChunkedStore request, it sends chunks in parallel using das_sendChunk, then once all chunks are sent uses das_commitChunkedStore to cause the data to be stored in the server and to retrieve the signed response to aggregate into the Data Availability Certificate. Server-side metrics are kept largely the same between chunked and non-chunked stores to minimize dashboard/alerting changes. In the context of chunked transfers, the metrics mean as follows: arb_das_rpc_store_requests - Count of initiated chunked transfers arb_das_rpc_store_success - Successful commits of chunked transfers arb_das_rpc_store_failure - Failure at any stage of the chunked transfer arb_das_rpc_store_bytes - Bytes committed arb_das_rpc_store_duration - Total duration of chunked transfer (ns) Additionally two new metrics have been added to count individual das_sendChunk requests: arb_das_rpc_sendchunk_success arb_das_rpc_sendchunk_failure
cla-bot
bot
added
the
s
This feature was unused and unmaintained, better to remove it for now since it complicates further das development.
anodar
previously approved these changes
Jun 3, 2024
…rvice Remove IterableStorageService from DAS
Remove IPFS das factory logic and stubs
joshuacolvin0
dismissed stale reviews from amsanghi, ganeshvanahalli, and anodar
via
b1edb9b
ganeshvanahalli
approved these changes
Jun 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add RPC for chunked send of DAS batches
This adds support for the batch poster in AnyTrust configuration to send batches to the DA committee in chunks of a configurable maximum HTTP POST body size. It adds new RPC methods to the daserver executable's RPC server, das_startChunkedStore, das_sendChunk, das_commitChunkedStore, which the clients created by the batch poster will automatically use if they are detected to be available on the committee server, otherwise it will fall back to the legacy das_store method. This allows an easy roll out of either the client or server first.
The payloads of the new RPC methods are all signed by the batch poster. As basic DoS prevention, at most 10 uncommitted stores can be outstanding, uncommitted stores expire after a minute, and a das_startChunkedStore with the same arguments is not replayable after a minute. The batch poster should only be trying to store one batch at a time, so this should be sufficient.
The new option --node.data-availability.rpc-aggregator.max-store-chunk-body-size is expressed in terms of the HTTP POST body size that the operator wants the chunk requests to stay under. 512B of padding is also added to whatever the user operator specifies here, since some proxies or endpoints may additionally count headers. This is orthogonal to settings like --node.batch-poster.max-size which control the maximum uncompressed batch size assembled by the batch poster. This should allow the batch poster to create very large batches which are broken up into small chunks to be sent to the committee servers.
Once the client has received confirmation to its das_startChunkedStore request, it sends chunks in parallel using das_sendChunk, then once all chunks are sent uses das_commitChunkedStore to cause the data to be stored in the server and to retrieve the signed response to aggregate into the Data Availability Certificate.
Server-side metrics are kept largely the same between chunked and non-chunked stores to minimize dashboard/alerting changes. In the context of chunked transfers, the metrics mean as follows:
arb_das_rpc_store_requests Count of initiated chunked transfers
arb_das_rpc_store_success Successful commits of chunked transfers
arb_das_rpc_store_failure Failure at any stage of the chunked transfer
arb_das_rpc_store_bytes Bytes committed
arb_das_rpc_store_duration Total duration of chunked transfer (ns)
Additionally two new metrics have been added to count individual das_sendChunk requests:
arb_das_rpc_sendchunk_success
arb_das_rpc_sendchunk_failure