non-blocking file upload API (formdata based)

[akosyakov]

• fix investigate how to use formidable to rewrite drag and drop upload in the navigator #4732 - one should be able to upload files via drag and drop without blocking UI and crashing the browser with out-of-memory.
• fix [quality] Move the upload service and its functionality to its own module #4923 - extract upload to own service
• fix Drag & Drop large file not copied when the file exceed 60M #969 - uploading a single file up to 2G is possible

It's breaking to redesign APIs. I've updated CHANGELOG with a record for it.

TODO:

• firefox
  • bogus percentage
  • often bad request - it's a Gitpod proxy issue, i don't get it locally
  • bogus progress styles
• linux
  • upload to the same device (cross-device link not permitted, i.e. from /tmp to /home)
• minimize sync fs access
• retry on network failures
• electron
  • remove upload
  • remove download

[lmcbout]

@akosyakov Question: When I Drag & Drop Theia into a workspace, there are many files , but I get the following error:

root ERROR Error: Upload Failed: Network Failure
at XMLHttpRequest.cb (http://localhost:3000/2.bundle.js:462:51)

I noticed that the UI did not freeze while transferring, The UI responded, I open a file and was able to navigate while the transfer occurred, but the Dropping files stopped at around 40%. Is there a limit for the number of files we can Drag & Drop ?

[akosyakov]

Is there a limit for the number of files we can Drag & Drop ?

@lmcbout No, it seems to be caused by network failure, so upload could not recover afterwards. I will try to reproduce.

[AlexTugarev]

Works nicely! 🎉
Thanks!

[akosyakov]

@lmcbout i was able to reproduce, unfortunately an http request is hanging after reconnection. I would try to timeout it in order to fail upload.

[akosyakov]

@lmcbout please try again, i've added a timeout of 5s which gets reset till the browser keeps sending upload progress reporting, so uploading of one big file still should be possible

[lmcbout]

@akosyakov I tried with your latest change and when I tried to upload Theia project to a new workspace and open files from another folder in my workspace, the following error occured:

root ERROR Error: Upload Failed: Network Failure
at rejectRequest_1 (http://localhost:3000/2.bundle.js:468:47)
at XMLHttpRequest.cb (http://localhost:3000/2.bundle.js:488:29)

[akosyakov]

@lmcbout it means that there are issues with an internet connection between you and the server. Where do you try? Do you have a proxy between you and the server with some limitations? Check in the chrome network tab why an upload request failed. There should be an http response from a proxy/server. You should see why it happens. I don't think it can be fixed in the code.

[akosyakov]

@lmcbout i cannot reproduce it. What is your local machine? Could it be that it is not powerful enough to handle upload and other UI actions in parallel in the backend, so that a browser does not receive a response from a local server?

[akosyakov]

@elaihau @vince-fugnitto Could you try as well? Upload theia and at the same time navigate through other files. I want to understand whether it is a general issue or an issue with a particular dev machine.

@lmcbout can you try in Gitpod?

[elaihau]

from my dev machine (ubuntu 16 + chrome), i tried (a few times) drag and drop theia folder into my workspace. Colors in Chrome became dark, and then Chrome crashed.

[akosyakov]

@elaihau Which chrome version? and if you do it Gitpod, the same?

[elaihau]

Version 73.0.3683.103 (Official Build) (64-bit)

[akosyakov]

My is 74.0.3729.131 (Official Build) (64-bit). It can crash if all files are loaded in memory, like the whole theia project, but code is chunking it with each chunk of 64Mb size. It can be that some version of a browser does not implement lazy file loading properly, so then it won't be possible to upload big amount of files. I am not even sure that there is a way to prevent a browser load everything into the memory then.

[lmcbout]

My Google version is: Version 72.0.3626.121 (Official Build) (64-bit)
It also crashed in my environment as well

[akosyakov]

@lmcbout i've increased a timeout to 30s and removed all sync access to fs during upload on the backend. Could you try please whether it makes the difference?

[akosyakov]

But in Gitpod all fine? If it is so then it won’t be a browser issue.

I was able to reproduce an issue with failed connection locally with Firefox. Browser gets very busy transferring files and does not give a chance to backend to process them, so connection gets unresponsive.

In practice you won’t upload to your local machine, so backend will stay responsive. It does not make much sense to test it locally.

Please check that uploading files up to 60 mb is still possible as on master. Preferably in Gitpod.

[akosyakov]

Also please use latest Chrome, yours are outdated.

[akosyakov]

@lmcbout I've added retry 5 times on network failures for each chunk.

[kittaakos]

WIth a local setup, I can upload a file from the workspace to the workspace. 🤦‍♂

I guess it runs into a loop because the Uploading Files... is showing 100% for a while now, and the console is full of:

at FileUploadService.<anonymous> (http://localhost:3000/2.bundle.js:476:51)
root WARN Error: Possible Emitter memory leak detected. 30 exit listeners added. Use event.maxListeners to increase limit
    at Emitter.../../packages/core/lib/common/event.js.Emitter.checkMaxListeners (http://localhost:3000/bundle.js:98865:26)
    at MutableToken._event.Object.assign.maxListeners (http://localhost:3000/bundle.js:98832:27)
    at FileUploadService.<anonymous> (http://localhost:3000/2.bundle.js:546:27)
    at step (http://localhost:3000/2.bundle.js:66:23)
    at Object.next (http://localhost:3000/2.bundle.js:47:53)
    at http://localhost:3000/2.bundle.js:41:71
    at new Promise (<anonymous>)
    at push.../../packages/filesystem/lib/browser/file-upload-service.js.__awaiter (http://localhost:3000/2.bundle.js:37:12)
    at FileUploadService.push.../../packages/filesystem/lib/browser/file-upload-service.js.FileUploadService.doSubmitForm (http://localhost:3000/2.bundle.js:510:16)
    at FileUploadService.<anonymous> (http://localhost:3000/2.bundle.js:476:51)

---

Another thing, super minor though, Upload Files... does nothing at all in electron. No errors, nothing. If it is not supported, we should consider disabling the command.

[kittaakos]

WIth a local setup, I can upload a file from the workspace to the workspace. 🤦‍♂

It turned out, it happens with a normal use case in both FF and Chrome.

[lmcbout]

@akosyakov My test was to add Theia into a folder by using Drag & Drop

I updated Chrome to : Version 74.0.3729.131 (Official Build) (64-bit)
The upload files was flickering between 0 to 1 %

I also tried on Firefox, I got a different behavior and at the end it stops after a few retry. It uploaded files up to 24% until it timed out. We have many retry now, but it seems we cannot upload a project like Theia into a workspace

When trying to load a 20 Meg file, Firefox and Chrome report a loop in the message from 0 to 100% and start over and over again, but never ends. See little video.

[akosyakov]

@lmcbout sorry, i introduced some bugs today, just pushed a fix for them, please try again, also @geropl @kittaakos

on FireFox the state is still pretty bad, requests are just sometimes bogus, i'm trying to understand why. Also performance of FireFox compare to Chrome is very bad at least on mac.

[elaihau]

I tested with the latest change on Chrome v74
and observed the following when drag and drop theia folder into my workspace:

• progress indicator showed the progress went from 0% to 46%, then
• status bar of theia became yellow, then
• message showed up a few times saying request timed out, will retry in x seconds, and finally
• a message showed up, saying the request timed out. at this point, theia status bar remained yellow, and chrome stopped responding (even in other opened tabs)

[akosyakov]

@elaihau locally or in Gitpod? Did you try a file of 60 Mb too?

[elaihau]

sorry for the confusion, i tested locally

[akosyakov]

@elaihau as i've explained local test does not make sense, see here: #5086 (comment)

I was able to reproduce an issue with failed connection locally with Firefox. Browser gets very busy transferring files and does not give a chance to backend to process them, so connection gets unresponsive.

In practice you won’t upload to your local machine, so backend will stay responsive. It does not make much sense to test it locally.

I will remove download/upload commands for electron.

[elaihau]

ok, i tested on gitpod, and saw almost the same behaviours. only difference is the "time out" messages started to show up at 26%.

[lmcbout]

@akosyakov
I also tested on a UNIX environment with Chrome:

• Big file : ~80 Meg and the transfer worked fine
• Project: Theia, I tried with Gitpod , created a new folder, then I tried to Drag & Drop " Theia " project to it. It transferred for a while, then it stops. But the bad thing here, my Chrome did not respond on any tabs after that, I had to kill all process associated to Chrome before being able to use it again

[akosyakov]

@elaihau @lmcbout I've found a memory leak in Chrome via request listeners, now they should be disposed when a request is done. Could you try again whether it makes a difference for you?

[elaihau]

This is related but probably belongs to another task & PR:

while files are being uploaded / downloaded, do we want to block users from switching from one git branch to another?

[elaihau]

Tested in gitpod. I had exactly the same problem that Jacques and I reported yesterday.

[kittaakos]

do we want to block users from switching from one git branch to another?

I do not think so.

[akosyakov]

while files are being uploaded / downloaded, do we want to block users from switching from one git branch to another?

I don't think so either, user can cancel it.

Related to issues:

• Unfortunately, I cannot reproduce Chrome issues, someone who can reproduce has to profile and see why Chrome does not release the memory. I need a snapshot after couple upload requests finished. @elaihau Could you do it please and upload here? Please don't forget to collect garbage before snapshot.
• In FireFox crashes are reproducible: although there is no memory leak in the page process, the main FireFox does not release memory with many upload requests. It looks like a bug in FireFox. I suggest we just limit amount of request to one here.

[lmcbout]

I tested the latest commit (e507a0b) with large files (80M and 120M) result: file transfered and able to open it, but the client stop responding for a short period (can see the bottom toolbar showing Offline) but it recovers.
Try multiple files (Theia project ) on Chrome and Firefox

• Chrome: same problem, freeze Chrome completely and need to restart chrome to access any tab view after
• Firefox, Stop with less data transfer, but the browser does not freeze, it stays available for using any tabs view after.

[akosyakov]

I am going to give up on FormData based implementation. Unfortunately some version of browsers on some platforms behave bogusly leading to memory leaks in main browser process overtime. Also since XMLHttpRequest does not allow streaming it has memory limitations for different browsers: not more than 2G at max.

I will go with chunking and streaming over the web socket, similar to https://github.com/mozilla/send. It will have very low memory footprint on both sides and allow to upload limitless amount of data. Hope i won't need to compromise UI responsiveness and upload speed for it too much.