Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add chunked mempool txids endpoint #73

Closed
wants to merge 1 commit into from

Conversation

mononaut
Copy link
Contributor

@mononaut mononaut commented Jan 25, 2024

The /mempool/txids endpoint returns every txid in the mempool, which can be slow enough to time out when there are a very large number of transactions.

This PR is one of two options for an alternative endpoint to allow the contents of the mempool to be queried in a series of smaller non-overlapping requests.

This option is a "chunked" endpoint /mempool/txids/<prefix>, where <prefix> is a hexadecimal string representing a big-endian prefix by which to filter the resulting txids.

The other option is a "paged" endpoint via PR #74.

For example, a call to /mempool/txids/f will return all txids in the mempool whose big-endian hexadecimal representation begins with "f".

Note that txids are normally represented in little-endian format, so the results would actually look like:

// fetch('/mempool/txids/f')
[ 
  "bf83ea15028b210a170bd6f44d5561bb8f070cc61a2bd50ac1873f210f6200f0",
  "0d703d9b9fd97efaff3e46dc4e6b1c3dfd858b9b5e9e862fdb6f89e6e4a501f0",
  "16e42701c6bfaf8d0b70518aae9228c78c1cf011ea8b28cac4464ac5f08202f0",
  ...
  "5389b806d49dd582a3d1cf356063d27aebb91df2e5a8694721353d258cfcffff"
]

The big-endian prefix approach is a bit unintuitive, but it matches how transactions are actually ordered internally which makes the queries more efficient.

In practice, I'd expect it to be used something like:

let all_txids = [];
for (let i = 0; i < 16; i++) {
  const result = await fetch(`https://mempool.space/api/mempool/txids/${i.toString(16)}`);
  const txids = await result.json();
  all_txids = all_txids.concat(txids);
}

@mononaut mononaut force-pushed the mononaut/chunked-txids branch from 5e28629 to d5f4c4c Compare January 25, 2024 20:52
Comment on lines +299 to +307
// get Txid range bounds for the given prefix
let start_bytes =
hex::decode(format!("{:0<64}", prefix)).chain_err(|| "invalid hash prefix")?;
let end_bytes =
hex::decode(format!("{:f<64}", prefix)).chain_err(|| "invalid hash prefix")?;
let start_txid =
Txid::from_hash(Hash::from_slice(&start_bytes).chain_err(|| "invalid hash prefix")?);
let end_txid =
Txid::from_hash(Hash::from_slice(&end_bytes).chain_err(|| "invalid hash prefix")?);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like there's a better way....... but it works. LGTM

@junderw
Copy link
Member

junderw commented Jan 26, 2024

I'm torn here...

chunking sounds good on paper, but the options are 1/16, 1/256, 1/4096... aka negative powers of 16. Which can easily fall into the "this one's too small but this one's too big..." category...

but it's so cool...

I'm leaning toward paging though.

@mononaut
Copy link
Contributor Author

yeah, I think paging is the way to go.

chunking would be worth it if txids were ordered in little-endian (since then this would also serve as an efficient txid prefix search, which would be pretty cool), but as it stands it's a bit too clumsy and unintuitive.

@mononaut
Copy link
Contributor Author

closing in favor of #74

@mononaut mononaut closed this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants