Duplicate Signature Being Returned When Using getSignaturesForAddress method on web3
#24417
Comments
|
That isn't the intended behavior but I'm not sure how this could be happening. Is your private RPC connected to bigtable for historical transaction queries? cc @CriesofCarrots |
|
As far as I know, it isn't. |
Do you happen to know if the block time for this result was still 1650132790 or was it something different? |
|
Sharing the full query and the unexpected response for that signature would be helpful for debugging |
|
Hi, so here's the entire details I have. Unfortunately, I don't have the exact times when these queries were made. Hopefully what I have is still of some value. Made a query to the RPC to get sigs with until value as Made the next query with until sig as Made the next query again with until value as |
|
For #24417 (comment), I don't unfortunately. All I know is that it happened again. Will run the tests again and record these to let you know. |
|
Ok, if you are able to reproduce again, please sure the results and we can take a look |
|
I think that I'm running into something similar to this. Sometimes, Anecdotally, this happens when I'm fetching transactions that are "near the tip" eg...polling for new transactions. I haven't seen this happen when I'm backfilling older transactions. I'm using the |
|
Real time follow up: I still experience the same thing using "finalized". |
|
To add to the comment on duplicates in backfilling, I think I've seen some duplicates in backfilling as well. They are rarer for sure but I do recall seeing some |
|
I've been able to reduce repro this in a small snippet. Here is a little bit of code that polls Raydium's address for new transactions. This loop will happily plug away and find new transactions as they come in and log the following output as expected: However, about once every 30 minutes, there's an anomaly, where a bunch of older transactions will get returned as follows. Notice the 1000 transaction results that all appear to be the same. |
|
For "until" requests, the RPC node will collect signatures until it hits the "until" signature. When your script gets the following response... The issue seems to be that the "until" signature is not found at all so the RPC node will just return the 1000 most recent signatures for an address. Note that the most recent block time for that response was "8:21:40 p.m.:130360512" which is actually not as recent as the block time from the previous response. So this leads me to believe that the your requests are being handled by two different RPC nodes and sometimes your request is handled by a node which hasn't finalized as recent of a block as the other node. I think the best recommendation I have for you is to ignore responses for data older than what you have already retrieved and retry later. Otherwise using a single RPC node should work ok |
|
Thank you! |
|
Thanks jstarry! that makes a lot of sense now. I'm pretty sure I'm just hitting a clustered endpoint. |
|
Happy to help diagnose the issue, I've created this issue to come up with a solution to avoid this problem: #24620 |
|
I'm also facing this issue, when hiting the same RPC (tried quicknode, Genesys Go, Ankr, etc). Also they're the most recent transactions, but hours old, so everything should be confirmed. What I've noticed, is that when this happens, the transactions belong to the same block. E.g. If I do a getSignaturesForAddress until Tx3, it returns Tx 1 and Tx 2 (correct). In my example, both Tx1, 2 and 3 are 5h old and I can reproduce this consistently. E.g. Transactions |
|
@arturferreira I looked up those transactions and your stated tx ordering is incorrect. I found that they were processed in the following order in block
I ran the following query for each signature: When searching until Lower slots and lower transaction indexes correspond to less recent transactions. The |
|
Hey @jstarry , Thanks for your response. I forgot to mention I'm using @solana/web3.js (am I on the wrong forum?) and I'm getting different results. I wrote a small snippet to prove it. I'm now using a 'before' parameter too so we can excluse newest transactions that were not there at the time of writing my first post, but the same (in my view) incorrect behaviour is happening. Maybe I'm missing something? ` const RPC_URL = 'https://ssc-dao.genesysgo.net/'; //Transaction in execution order let optsArr = {until: '5zCk4r61KrxS6wrVevGfM9Si4E4mStcNWp5fdYVKH1dR2Qia59sLy5Rf5WHXrZ9kYhjHiVcEVnqFzeoK1AiUN72P', before: '7dAUj7oMd8pfepdZ4DXUUJfhYpWAjK1oZwdVrJknTp9AWYSUfgpPf2PLJad37AYrBAm8vgT5dJChjztGV1YeH3S'} optsArr = {until: '3UjqKvrNDaDexj41c4fQmmfFMkDeEZox2XNWr1bkuhcsbBiKSgsCoQicf8SaQq2Vk9wwPha4PiGuWvsKhQpZuzbF', before: '7dAUj7oMd8pfepdZ4DXUUJfhYpWAjK1oZwdVrJknTp9AWYSUfgpPf2PLJad37AYrBAm8vgT5dJChjztGV1YeH3S'} The code above outputs the following:
Can you please confirm I'm not missing anything? Thanks, Edit: I get the correct results when using https://api.mainnet-beta.solana.com, so this seems to be an RPC problem, though I get it with Quicknode, Ankr and Genesys Go, just not with https://api.mainnet-beta.solana.com |
|
Hmm yeah it must be an RPC problem, can you bring it up with them to debug? |
|
@jstarry GenesysGo here. The only thing we can think of is there's an edge case with how transactions are filtered from local storage only when Google Bigtable is configured on the node AND when the transaction is still available in the node's local storage. After letting our entire network sit for a day completely untouched, the example curls referenced above now return the same result as what's returned from api.mainnet-beta.solana.com. Feels like a ghost in the system with some very specific conditions and obviously it's a shared issue as it's happening with other RPC providers that I'd assume also have Google Bigtable configured. That's the only difference I can think of between these networks and api.mainnet-beta.solana.com. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
I've an internal project which makes requests to a private Solana RPC to get transaction signatures every few seconds. Every poll, I get new signatures with
getSignaturesForAddress, sort them in descending order by blocktime (or by slot if blocktime is null). In the next poll, I make the query for getSignaturesForAddress again with until parameter set as the signature with highest block time from the previous poll. This should return me new signatures which have happened until that signature.Now, multiple signatures can have the same blocktime. Does web3 guarantee that those signatures with the same blocktime are not returned?
We've been seeing duplicate signatures in next polls with this approach. Here's the last sample run where duplicates happened:
Is this a bug or is this the intended behaviour? Any explanation would be very appreciated!
The text was updated successfully, but these errors were encountered: