-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add blob download endpoint (getBlobs) #286
Conversation
Co-authored-by: 0xGabi <[email protected]> Co-authored-by: Paul Harris <[email protected]>
From what I can see the request / response structures look similar, just some differences in paths. I think a combination of the two makes sense to me (e.g.
I'm still pretty new to this so I might be missing something - keen to hear suggestions / opinions. |
One potential issue of note: in the underlying libp2p protocol, we don't support fetching finalized items by root, instead only supporting by-slot lookups - it might be prudent to note this in this spec as well, that sidecar retrieval for finalized blocks should by done by slot else the underlying index might not be available. Doing it this way allows clients to implement this API using linear storage which is cheaper and more efficient and more closely matches the requirements that clients have to support in order to implement the mandatory parts of the protocol. |
I'd be inclined to go for just |
Added Prysm implementation here: prysmaticlabs/prysm@50b672a |
@arnetheduck thanks for this! I don't fully understand the implication of this on the Beacon API, would appreciate if you could elaborate a bit more - if the blob is already available in CL storage and p2p lookup isn't required, would this still be an issue? |
in the core protocol, there is no requirement to index finalized data by hash, only by slot - this applies equally to blobs as it does to blocks. Because finalized data is strictly linear, the index-by-hash is not needed and looking things up by hash is comparatively inefficient, as is building and maintaining the hash index itself. "CL storage" for finalized data can be implemented as a flat file to serve core protocol needs - by suggesting consumers to use slots, the same efficiency can be had in the REST API as well, which saves on disk space, processing and complexity, in the CL. edit: you're indeed right that it is also more simple to forward a request to the network when topping up local storage if that request is done by slot, since the network doesn't carry said hash index. |
Good point on |
beacon-node-oapi.yaml
Outdated
@@ -95,6 +95,8 @@ paths: | |||
$ref: "./apis/beacon/blocks/root.yaml" | |||
/eth/v1/beacon/blocks/{block_id}/attestations: | |||
$ref: "./apis/beacon/blocks/attestations.yaml" | |||
/eth/v1/beacon/blobs_sidecars/{block_id}: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd actually be inclined to make this the same pattern as attestations, so the path would be more like
/eth/v1/beacon/blocks/{block_id}/blobs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't that imply that the blobs are part of the block structure, though?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to /eth/v1/beacon/blobs/{block_id}
We should add to top level |
Consensus-spec references may need to be fixed once deneb is renamed in that project. related to ethereum/consensus-specs#3207
@rolfyone @mcdee @arnetheduck thanks a lot for the explanation, If I understand you correctly, you're suggesting to not support lookup by root for blobs (and perhaps blocks too) as it would allow clients implement this more efficiently. I made the assumption we would want the same lookup (block id) for blobs as well, as looking up by root is already supported for blocks in the Beacon API. I don't really understand the libp2p protocol and storage implementations enough to make a comment about this (but interested to find out more if there's any resources you could point me to) - I'll wait for others to chime in on this. |
Perhaps more weakly than this, ie "recommend" that consumers look up by slot - it's possible that some clients build a by-root index for the entire history (consider archive nodes for example) but it is certainly more "compatible" to not rely on it and instead use |
…this back to bellatrix transactions like the other similar references.
Just going through the changes in the "Free the blobs" PR ethereum/consensus-specs#3244, and will soon be updating this branch accordingly. |
I don't think this will have to change too much, I think it should probably return the same struct we return in the |
beacon-node-oapi.yaml
Outdated
@@ -95,6 +95,8 @@ paths: | |||
$ref: "./apis/beacon/blocks/root.yaml" | |||
/eth/v1/beacon/blocks/{block_id}/attestations: | |||
$ref: "./apis/beacon/blocks/attestations.yaml" | |||
/eth/v1/beacon/blobs_sidecars/{block_id}: | |||
$ref: "./apis/beacon/blobs_sidecars/blobs_sidecar.yaml" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do both need to be plurals? is blob_sidecars
accurate?
if the paths / files / object names can be consistent that'll make it easier to navigate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm I think this is on an old commit as well, I just did a global search for blobs_
and not finding anything. I'll make sure the paths / file name are matching if we end up changing the paths 👍
So i think this is probably close, just whether namings are correct also my changes markdown comment about whatever the path ends up as :) Then I think we should be right to merge... |
Hope for one last merge to fix conflicts... |
Thanks @rolfyone I've merged master into this branch and updated the paths. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
type: object | ||
properties: | ||
data: | ||
type: array |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We appear to have the type BlobSidecars
that is an array of blob sidecars with the same parameters as this. Should that be used in place of redefining the array here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Updated in 096aadc, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks team!
Description
Addresses #282. Add
/eth/v1/beacon/blobs/{block_id}
endpoint to support retrieval of blobs by block id.