Relax error propagation for batch loading mempool txs #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
junderw
reviewed
Nov 13, 2023
| @@ -506,13 +538,12 @@ impl Daemon { | |||
| .iter() | |||
| .map(|txhash| json!([txhash.to_hex(), /*verbose=*/ false])) | |||
| .collect(); | |||
|
|
|||
| let values = self.requests("getrawtransaction", ¶ms_list)?; | |||
| let values = self.retry_request_batch("getrawtransaction", ¶ms_list, 0.25)?; | |||
There was a problem hiding this comment.
- Magic number / arbitrary value.
- This doesn't feel like it should be set here.
At the very least we need a TODO to properly bubble this up the call stack and make it into a startup config param or something.
junderw
approved these changes
Nov 13, 2023
junderw
reviewed
Nov 13, 2023
| Ok(parsed_reply) => results.push(parsed_reply), | ||
| Err(e) => { | ||
| failed_requests += 1; | ||
| warn!( |
There was a problem hiding this comment.
Maybe lower to debug level... it is expected to be frequent...
wiz
approved these changes
Nov 13, 2023
SatoKentaNayoro
pushed a commit
to boolnetwork/mempool-electrs
that referenced
this pull request
Nov 29, 2024
…atching Relax error propagation for batch loading mempool txs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the initial batched mempool load takes too long, some of the transactions are likely to have been removed from the mempool (replaced or evicted) by the time we request them.
Currently, any missing transaction in the whole batch results in an error that propagates up to fail the entire gettransactions result, which can make it almost impossible to make progress.
This PR allows individual transaction requests in each batch RPC chunk to fail without failing the whole request. Those failed txs are simply filtered out of the response, and can just be fetched on the next mempool update (if they still exist).
If more than 25% of requests fail, we do still abort the request, and bubble up the last error.
This required relaxing the
assert_eq!(txhashes.len(), txs.len())assertion ingettransactions, and I suppose it could potentially also cause issues with missing parents etc, so I'm not really sure how safe this is 🫤