state.buffer[0].length Cannot read properties of null (reading 'length')

[gal-checksum]

Environment details

• OS: Mac 13.5 (22G74)
• Node.js version: 21.2.0
• npm version: 10.2.3
• @google-cloud/storage version: 7.6.0

Getting the following error when trying to download a file:

source-map-support.js:496 node_modules/stream-shift/index.js:16
    return state.buffer[0].length
  source-map-support.js:499 TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (node_modules/stream-shift/index.js:16:28)
    at shift (node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:519:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

This only happens when I download the file from a certain project. If I copy the exact same config and code but initiate it within a script, it works

[unifractal]

I am seeing the same problem. But for me it seems to be an issue with most versions, not just 7.6.0. I have tested all the way back to 6.12 and it is broken for me

[gal-checksum]

Has to do with this:
mafintosh/stream-shift#6

Downgrade your node.js version to below 21.2

[ddelgrosso1]

@unifractal can you confirm you were also using node 21.2+?

[PatricioPoncini]

Hi, I have the same problem. I cannot upload files larger than approximately 70KB. Does it work if I downgrade my Node.js version, or is it a library issue?

[unifractal]

Yes, i can confirm. I was also on node 21.2+ and downgrading to e.g. Node version LTS 18 fixed my problems.

Definitely a problem caused when upgrading node. It just sucks when it happens automatically using e.g. Google Cloud run 😂

[Idnan]

I was also having the same issue with NodeJS 21.2+ and downgrading to NodeJS 21.1+ fixed the issue.

[PatricioPoncini]

It doesn't work for me, even after downgrading Node.js to 21.1. I'm using Bun. Could it be an issue with Bun and this library? I posted a question; it's my first time using this library with Bun. My problem is when I want to download a file from my bucket

#2370

[ddelgrosso1]

For those having problems on Node 21.2 it appears the solution is to downgrade to get around the problem. Looking at the provided stack trace it appears to mainly be coming from Node internal libs / Duplexify. It might be worth opening an issue for the Duplexify folks to see if they can resolve it on their side. We do intend to remove Duplexify from this library in the future (#2041) However, that will not happen until the next major revision to the library which is likely sometime next year.

[jontybrook]

Ditto this. Downgrading to node v20.10 resolved fixed it.

[drichardson]

FYI I ran into this with current LTS of node (20.11.0). I worked around it by removing the {destination: my_output} option from the download function and writing it to a file myself.

[mpalmerlee]

I experienced the same issue with Node 20 although my error message was slightly different:

TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (/prod/api/node_modules/.pnpm/[email protected]/node_modules/stream-shift/index.js:16:28)
    at shift (/prod/api/node_modules/.pnpm/[email protected]/node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (/prod/api/node_modules/.pnpm/[email protected]/node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (/prod/api/node_modules/.pnpm/[email protected]/node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:518:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

To work around I changed my code from:
await this.storage.bucket(config.gcp.storageBucket).file(gcsPath).download(options);

To:
await this.transferManager.downloadFileInChunks(gcsPath, options);

Like in this example downloadFileInChunksWithTransferManager.js

[russell-liu-splyt]

stream-shift 1.0.2 works fine with node 20.11.0

[mksony]

stream-shift 1.0.2 works fine with node 20.11.0

Can confirm that 💯 we had to add an override to our package.json to enforce stream-shift 1.0.2.

[MilosLukicBM]

Can also confirm that setting stream-shift to 1.0.2 for all dependencies (used the overrides method) works with node 20.11.0 🎉

[MattBidewell]

Those who might end up here whilst running a Google Cloud Function, theres a chance the underlying image of your cloud function has been updated and this bug has manifested itself!

For the time being, you can add the following to your package.json to get around the issue the storage API is updated:

JSON

  "overrides": {
    "stream-shift": "1.0.2"
  }

Worth noting that stream-shift is a dependency of duplexify which appears unmaintained...

[majikandy]

Can also confirm this works in the package json to override to 1.0.2 using the overrides block above.

Using the functions-framework to run the function locally doesn't cause the stream-shift TypeError: Cannot read properties of null (reading 'length'), it only seems to happen when it runs in the cloud so we only became aware on a rebuild/deploy.

[danielduhh]

Thanks for the investigation!
Please upgrade to stream-shift 1.0.2 (a dependency of duplexify in this repo). This will fix the error. We have plans to move away from duplexify altogether -- you can track that here: #2041

[npomfret]

Had the problem with node v20.11, downgraded to v20.10 and the problem went away. Thank you.

[undesicimo]

downgrading to 20.10 seems to work for now