Premature close error when fetching 25 or more accounts at once with getAccountInfo

[peroxy]

Overview

Using the Connection class' method getAccountInfo with Promise.all and at least 25 requests will fail with:

Error: failed to get info about account Cfuy5T6osdazUeLego5LFycBQebm9PP3H7VNdCndXXEN: **FetchError: Invalid response body while trying to fetch <redacted RPC url> Premature close**
    at Connection.getAccountInfo (/home/dev/sandbox/test/node_modules/@solana/web3.js/src/connection.ts:3517:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 12)

I understand this is inefficient; ideally, I should use getMultipleAccounts instead, but that is not always possible. I am using multiple SDKs from different protocols that do not use getMutlipleAccounts method and I have to rely on fetching multiple accounts with Promise.all concurrency to make it faster. We use a non-rate-limited RPC in our backend applications so it should not be an issue.

I have also tried to fetch different account sizes, the one I used in this example has 4064 bytes, but I could reproduce the same error with 250/650/2000 byte account sizes.

Steps to reproduce

Using "@solana/web3.js": "1.75.0" version (also fails on 1.73.0, haven't tested on others):

import {Connection, PublicKey} from "@solana/web3.js";

const connection = new Connection(
    "https://api.mainnet-beta.solana.com");

const promises = [];
for (let i = 0; i < 26; i++) {
    promises.push(connection.getAccountInfo(new PublicKey('Cfuy5T6osdazUeLego5LFycBQebm9PP3H7VNdCndXXEN')));
}
Promise.all(promises).then(x => console.log(x.length)); // <-- this call will fail with the error described

To reproduce the error you can clone this repo: https://github.com/peroxy/solana-web3.js-premature-close-bug

and run commands:

    yarn install
    yarn start

Description of bug

I would expect this to work correctly and return all the account infos instead of failing with a very cryptic error.

[steveluscher]

I understand this is inefficient; ideally, I should use getMultipleAccounts instead…

Not true! When you use http2 pipelining you only open one connection to the server and you allow the RPC to load balance in a way that makes sense for their cluster. Also, you get the incredible benefit of being able to stream the results in and be able to recover the rest of the results when some of them fail (if you use Promise.allSettled).

[steveluscher]

I just tried the repro above and it went off without a hitch. Are you still experiencing this?

[peroxy]

I just tried the repro above and it went off without a hitch. Are you still experiencing this?

Did you try it in my repository?

git clone [email protected]:peroxy/solana-web3.js-premature-close-bug.git
yarn install
yarn start

I've just tried and reproduced the issue with the commands above. I've also tried updating to the latest web3.js version and it still did not work.

[steveluscher]

OK, update. It looks like this started with [email protected] specifically. Version 2.6.7 works fine.

[steveluscher]

FetchError: Invalid response body while trying to fetch https://api.mainnet-beta.solana.com/: Premature close
    at Gunzip.<anonymous> (/home/sol/src/solana-web3.js-premature-close-bug/node_modules/.pnpm/[email protected]/node_modules/node-fetch/lib/index.js:400:12)
    at Gunzip.emit (node:events:525:35)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ERR_STREAM_PREMATURE_CLOSE',
  code: 'ERR_STREAM_PREMATURE_CLOSE'
}

[steveluscher]

OK. Something is up with the Triton RPCs. I'll follow up. If I replace the RPC with one from Helius, it works. cc/ @linuskendall.

[peroxy]

OK. Something is up with the Triton RPCs. I'll follow up. If I replace the RPC with one from Helius, it works. cc/ @linuskendall.

Interesting. I recall trying this out with Triton + Lido RPCs and it did not work with both of them, but maybe I just got unlucky.

[steveluscher]

From the release notes for [email protected]:

For responses using chunked Transfer-Encoding without a Content-Length, this adds a check at the socket's close event to verify that the last bytes received were the final chunk bytes, i.e. 0\r\n. If not, it creates a Premature close error and sends it to response.body.destroy()

So yeah, that's almost certainly what's happening here. Triton RPCs don't supply Content-Length so they're probably tripping this error.

[steveluscher]

Triton is now aware of this. I'm going to close this issue since there's nothing we can do from the library, other than to try to lock to an old version of node-fetch (which we can barely enforce). Thanks for the report!

[steveluscher]

Interestingly this does not fail with Node 16. Does fail with Node 18 or 20.

[github-actions]

Because there has been no activity on this issue for 7 days since it was closed, it has been automatically locked. Please open a new issue if it requires a follow up.