Uploading big files crush page in Chrome

[mattermoran]

So I'm doing as in this example https://github.com/ipfs/js-ipfs-api/tree/master/examples/upload-file-via-browser and everything works fine. But when I try to upload files that more than 3gb page crushes.

[kostysh]

I am facing the same issue

[julientregoat]

I have the same issue, except with files over 50-100mb crashing it. I've tried ReadableStream, pull stream, buffer instances... nothing working. Smaller files get there just fine. Running node 8.10. Thought pull streams would've worked based on this

[daviddias]

@julientregoat -- ipfs/js-ipfs#952 -- says that things are good uploading over 1GB on js-ipfs. Here, you are using js-ipfs-api, the HTTP Client library and talking with a go-ipfs daemon, it is a different scenario.

This should just be a buffering/streaming issue when creating the HTTP request. If anyone is available to tackle this, it would be great to have it solved.

[julientregoat]

@diasdavid ah got it! sorry, I didn't understand. still new to this but will try to play with it to see if I can get anything going.

[channprj]

It works on Safari. Just crashes in Chrome. How can I handle this? Does anybody work on it?

[hacdias]

After some discussion on IRC, @alanshaw @lidel and I found out that this happens because the library we use to shim the native Node.js http module, stream-http, stores the whole request into a Buffer and only then it sends it. That's why it uses so much memory with big files.

There seems to be an experimental Streams API for browsers, but it doesn't seem to be very supported right now.

Maybe we'll need to wait or find another way to send the files in smaller chunks.

[daviddias]