Making blob.stream() more stream friendly

[jimmywarting]

Version

17.5

Platform

mac

Subsystem

No response

What steps will reproduce the bug?

const blob = new Blob(data)

// Sometime later
blob.stream()

How often does it reproduce? Is there a required condition?

it reproduce every time, it's not clear to everyone that stream() will internally use blob.arrayBuffer()

What is the expected behavior?

to yield small chunks of data to not allocate as much memory as blob.size

What do you see instead?

I see memory spikes that when i create large blobs and then later tries to read them that memory goes way up.
this is one of the reason we are sticking to using fetch-blob instead of node's own built in blob implementation instead. I wish i did not have to use this package anymore and that i could also take adv of transfering blob's over workers with the built in transferable blob.

Additional information

No response

[meixg]

later tries to read them that memory goes way up

There used to be an array copy while reading blob stream, maybe this is the reason. Can you try on the master branch?

---

There is still a full copy when creating a stream though.

[jimmywarting]

Here is where it reads the hole blob into arrayBuffer before it starts streaming the data in chunks

node/lib/internal/blob.js

Lines 325 to 328 in a7164fd

	async start() {
	this[kState] = await self.arrayBuffer();
	this[kIndex] = 0;
	},

[jimmywarting]

the pull should rather try to fetch the needed chunked arrayBuffer()

maybe do something like this psudo code?

pull(ctrl) {
  slice = await this.slice(start, end).arrayBuffer()
  ctrl.enqueue(slice)
}

[jimmywarting]

it's somewhat unexpected that all chunks use the same underlying buffer and that each chunk has a byte offset.

chunks[0].buffer === chunks[1].buffer

// Should rather be false
chunks[0].buffer !== chunks[1].buffer
chunks[0].byteOffset === 0

[jasnell]

I think the work that @flakey5 is looking to do in #45188 should address this.

The initial implementation of Blob was meant to be as simple as possible to get something working. The intent was always to revisit it to implement performance optimizations once it was in use.