chore: review proof routes #6843
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nflaig
commented
Jun 3, 2024
| @@ -8,8 +8,7 @@ import {ArrayOf} from "../../utils/codecs.js"; | |||
| import {VersionCodec, VersionMeta} from "../../utils/metadata.js"; | |||
|
|
|||
| export const CompactMultiProofType = new ContainerType({ | |||
| // TODO ensure limit of all lists is sane | |||
| leaves: ArrayOf(ssz.Root), | |||
| leaves: ArrayOf(ssz.Root, 10000), | |||
There was a problem hiding this comment.
I am asuming this is sufficient and sane enough, same value we use in
but I am not that familiar with return values of proof routes, would be good to get eyes on this. Better to have a too high limit, than breaking large proofs by setting it too low.
There was a problem hiding this comment.
10k is plenty big, I wanna say we had 512 set as MAX_PROOF_GINDICES at some point.
There was a problem hiding this comment.
we have a configurable limit on the server which as you noted is 512. the limit in the ssz type might still be useful to avoid crashing the client
lodestar/packages/beacon-node/src/api/impl/proof/index.ts
Lines 13 to 15 in f21fc3c
preventing server dos vector should be covered
| // getStateProof: { | ||
| // ...serverRoutes.getStateProof, | ||
| // handler: async (req) => { | ||
| // const args = definitions.getStateProof.req.parseReq(req); |
There was a problem hiding this comment.
we could implement the same logic in serialize / deserialize in the route without overriding the handler if that's the format we wanna keep
11 tasks
wemeetagain
approved these changes
Jun 3, 2024
wemeetagain
added a commit
that referenced
this pull request
Jun 10, 2024
* Add config route definitions * Add debug route definitions * Add events route description * Add lightclient route definitions * Flatten function params * Type safety for optional params in write / parse req * Method args are optional if only optional props * Fix genesisValidatorsRoot type issue * Revert requiring all params in write / parse req * Update http client errors * Add lodestar route definitions * Add node route definitions * Add proof route definitions * Add builder route definitions * Add validator route definitions * Application method response can be void * Generic options can be passed to application methods * Default endpoint request type has body property * Improve types of transform methods * Export server types from index (to be removed) * Update config api impl * Update lightclient api impl * Update events api impl * Update lodestar api impl * Update proof api impl * Update node api impl * Update debug api impl * Update state api impl * Update pool api impl * Update blocks api impl * Partially update validator api impl * Update beacon routes export * Align submitPoolBlsToExecutionChange method args * Filters are always a object * Update errors messages * Add beacon client methods * Add missing routeId label to stream time metric * Fix json casing in codecs * Apply remaining changes from #6227 * Produce block apis only have version meta * Add block values meta to all produce block apis * Apply changes from #6337 * Handle unsafe version in WithMeta and WithVersion * Restore server api error * Update fastify route types * Update server routes / handlers * Remove unnecessary type cast * Restore per route clients * Fix beacon route types * Remove option to patch fetch from http client * Update eventstream client, remove fetch override Fallback does not work like this, see #6180 for proper solution * Use StringType for validator status until #6059 * Remove empty fetch.ts file * Add a few todos * Update builder client and server routes * Update beacon exports * Update api index exports * Update builder index imports * Improve type safety of schema definitions * Add headers to fastify schema * Fix schema definition type * Add missing schemas to route definitions * Fix response codec type * Remove response codec type casts * Fix casing in json only codec * Reuse EmptyResponseCodec * Update base rest api server * Update keymanager routes, client and server * Reuse data types in keymanager impl * Do not await setting headers, not a promise * Improve type safety of empty codecs * Only require to implement supported req methods * Handle requests that only support one format * Handle responses that only support one format * Add json / ssz only req codecs * Update only support errors * Fix assertion * Set correct accept header if only supports one format * Fix eslint / prettier issues * More formatting fixes * Fix fallback request retries in case of http errors * Formatting of res.error * Add add retry functionality to http client (from #6387) * Update rewards routes and server (#6178 and #6260) * Allow to omit body in ssz req if not defined * Always set metadata headers in response * Cache wire format in api response * Only call raw body for json meta * Update api package tests (wip) * Test json and ssz format in generic server tests * Add a bunch of todos * Fix a few broken route definitions * Fix partial config test * Another todo note * Stringify body of json requests * Override default response json method * Validate external consensus version headers in request * Add error handling todo * Skip body schema validation for ssz request bodies * Clean up generic server tests * Pass node, proof, events generic tests * Use enum for media types * Fix a bunch of route definitions * Add justified to blockid type * Properly handle booleans, remove block values codec * Create Uint8Array test data without allocating Buffer * Let fastify handle Buffer conversion * Convert Buffer to Uint8Array in content type parser * Fix build issues * Fix fork type in builder routes * Add some notes * Properly parse request headers * Fix incorrect type assumptions in transform * Generic server tests are passing (except lightclient) * Correctly handle APIs with empty responses * Update getHeader return type to reflect no bid responses * Do not append '?' to URL if query string is empty * Let server handler set status code for parsing errors * Remove unused import * Rename function, request specific * Completely drop ssz support from getSpec * Spec tests are passing against latest releases * Drop unused fastify route config * Drop ssz request from builder routes, not yet supported * Remove import * Apply change from #6695 * Update execution optimistic meta * Apply changes from #6645 * Add workaround to fix epoch committees type issue * Add todo to fix inefficient state conversion * Convert committee to normal array * Apply changes from #6655 * Align args of validators endpoints * Convert indices to str in rewards apis * Update api spec version of README badges * Revert table formatting changes * Make this accessible for class-basd API implementations * Throw err if metadata is accessed for failed response * Add assertOk to api response * Tweak api error message * Update operationIds match spec value * Add missing version to blob sidecars metadata * Test headers and ssz bodies against spec * Minor reordering of code in spec parsing * submitBlindedBlock throws err if fork is not execution * responseOk might be undefined * Remove statusOk from route definition * Remove stale comment * Less build errors in beacon-node * getBlobSidecars return version from server impl * Update validator produce block impl * More expressive pool method args * Application methods might be undefined in mock implementations * Adress open TODOs in server handler * Api response methods are synchronous now * Fix all remaining build issues * Use more performant from/toHex in server api impls * Clean up some TODOs * Fix ApiError type * Errors related to parsing return a 400 status code * Simplify method binding * Forward api context to application methods * There is no easy way to make generic opts typesafe * Better separation of server / client code * Fix comment about missing builder bid * Remove todo, not worth the change / extra indentation * Rename route definitions functions * Return 400 if data passed to keymanager is invalid * Properly handle response metadata headers * Fix lint issues * Add header jsdoc * Move metadata related code into separate file * Remove ssz from POST requests without body * Only set content-type header if body exists * Fix headers extra * POST requests without body are handled similar to GET requests * Fix http client options tests * Improve validation and type safety of JSON metadata * Add type guard for request without body * Differentiate based on body instead of GET vs POST * More renaming * Simplify RequestCode type * Review routes, improve validation * Remaining local diff * Fix accept header handling if only support one wire format * Update 406 error to more closely match spec example * Enforce version header via custom check instead of schema * Use ssz as default request wire format * Log failure to receive head event to verbose * Do not set default value for context * Update getClient return type to better align with method name * Consistent pattern to get route definitions * Dedupe api client type for builder and keymanager * Fix fallback logic if server returns http error * Update head event error logging * Retry 415 errors with JSON and cache SSZ not supported * Use fetch spy to assert call times * Update comment * Update getLightClientUpdatesByRange endpoint meta * Do not forward ssz bytes of blinded block to publishBlock * Fix lightclient e2e tests * Version header in publishBlock api is optional * Reduce type duplication * Add option to override request init in route definition * Add JsonOnlyResp codec * Validate boolean str value from headers * Document default wire formats * Simplify merging of inits in http client * Remove type hacks from fetchBeaconHealth * Reduce call stack in http client * Add .ssz() equivalent method for json to api response * More http client tests * Ensure topics query is provided to eventstream api * Validate request content type in handler Fastify does not cover all edge cases * Review routes, fix param docs, no empty comments * Fix typo * Add note about builder spec not supporting ssz * Consistently move keymanager jsdoc to routes * Sanitize user provided init values before merging * Remove unused ssz only codec * Allow passing wire formats as string literals * chore: review proof routes (#6843) Review proof routes * chore: review lightclient routes (#6842) Review lightclient routes * chore: review node routes (#6844) Review node routes * feat: add cli flags to configure http wire format (#6840) * Review PR, mostly cosmetic changes * Fix event stream error handling --------- Co-authored-by: Cayman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Pulled out of #6749 to discuss changes separately and for easier review.
Description
We broke (in terms of a breaking change) the ssz response of proof routes and it is no longer aligned with spec proposal ethereum/beacon-APIs#267 (which did not get merged).
This PR is mostly just cleanup and to confirm that we are fine with doing a breaking change to the route. As it is not spec'd and has very limited amount of users it might not be relevant, could still add a note in the release this gets included.