[HTML5] Replace XMLHttpRequest(s) with Fetch.

[Faless]

In this PR:

• The Preloader in the Engine support code now uses Fetch instead of XMLHttpRequest.
• By using fetch to load the wasm file, we can now use steaming instantiation which should improve time to first interaction Godot HTML5 games have a really long time to first interaction #41118 .
• The exporter now also sets the expected wasm and pck size during export, to help get a more accurate loading bar.
• HTTPClient now uses Fetch too, this allows chunked body retrieval (e.g. for event/stream, or in general, streamed reading), among other improvements. See also [BUG] No updates from Real Time Database in HTML5  GodotNuts/GodotFirebase#130 .

There is a 3.2 version of this here: https://github.com/Faless/godot/tree/js/3.x_fetch_world .

[akien-mga]

Thanks!

[jjcampis]

hi, how can i revert the fetch to xhr in 3.3?

I need this because im exporting html and embedding in a webview in android...
this is because i need bluetooth feature and i print whatever i want to send in webview console and then i send whats in the console..

the 3.3.2 godot version has the virtual keyboard which works!

but i cant load the html in the webview with fetch BUT with xhr work like charm

[Calinou]

hi, how can i revert the fetch to xhr in 3.3?

There is no way to revert to XHR, except by modifying the source code and compiling custom HTML5 export templates. fetch() is considered to be the way forward.

[Faless]

@T3chnoT00l5 you can try adding a polyfill in the Head Include section of the export

[jjcampis]

No success! :(
I need xhr because the index.html .wasm .pck is in an assets folder inside a webview of android
i need this way because the app reads what i print on console and send it via bluetooth

the fetch method doesnt allow me to load the wasm and pck because fecth API doesnt support the "file://" protocol

is there any way to modify the index.js or something that allow read the .pck and .wasm via xhr?

[Calinou]

I need xhr because the index.html .wasm .pck is in an assets folder inside a webview of android

Out of curiosity, why aren't you using Godot's Android export instead?

[jjcampis]

I need xhr because the index.html .wasm .pck is in an assets folder inside a webview of android

Out of curiosity, why aren't you using Godot's Android export instead?

Because i need to send data via Bluetooth so when i press some buttons in the godot ui which Is in a webview the android app reads the console of the webview and send the text vía Bluetooth... Godot doesn't provide Bluetooth comunication so i've done this workaround....

Godot 3.2 exports works like charm BUT when i click on LineEdit the virtualKeyboard doesn't show up...

In 3.3 there Is a virtual ketyboard experimental that works fine for me but the Fetch method doesn't allow to get the files.... A total shame

[jjcampis]

I Hope in godot 4.0 we can Select which method use to load the .wasm and .pck

Will be really appreciated :)

[lloesche]

Question: when using Fetch the browser now adds its Accept-Encoding: gzip, deflat, br header to all my client requests. When the server then correctly responds with a brotli compressed answer what is a Godot game supposed to do with that data? Implement its own brotli decompression in GDScript? What is the correct way to handle these compressed responses? Or rather why does the client even receive the compressed response instead of a decompressed one? After all the browser added the Accept-Encoding header. I feel like it should now make sure to correctly handle that encoding and give me uncompressed data?

[Calinou]

Question: when using Fetch the browser now adds its Accept-Encoding: gzip, deflat, br header to all my client requests. When the server then correctly responds with a brotli compressed answer what is a Godot game supposed to do with that data? Implement its own brotli decompression in GDScript? What is the correct way to handle these compressed responses? Or rather why does the client even receive the compressed response instead of a decompressed one? After all the browser added the Accept-Encoding header. I feel like it should now make sure to correctly handle that encoding and give me uncompressed data?

Godot would need to have a Brotli decoder implementated, but this will increase the binary size and build complexity as Google's Brotli library will need to be linked against. Feel free to open a proposal for this 🙂

Implementing Brotli decompression in GDScript is technically feasible, but I wouldn't be wishing it on anybody to do…

[lloesche]

So what's the answer? The http request gets augmented with Accept-Encoding headers by the browser which cause the server to compress the response. But then there's nothing that can handle the compressed response? I feel like there's something missing.

[Calinou]

So what's the answer? The http request gets augmented with Accept-Encoding headers by the browser which cause the server to compress the response. But then there's nothing that can handle the compressed response? I feel like there's something missing.

We unfortunately have no control over the headers added to fetch() by the browser. Therefore, we pretty much have to add a Brotli decoder implementation in Godot to make Brotli-compressed responses usable in Godot.

In the meantime, you can host a proxy server that only sends uncompressed responses (or gzip-compressed responses in master).

[lloesche]

We unfortunately have no control over the headers added to fetch() by the browser. Therefore, we pretty much have to add a Brotli decoder implementation in Godot to make Brotli-compressed responses usable in Godot.

Interesting, and there is no way to have the browser handle decompression of the response? I know nothing about JavaScript, I just can't imagine that JavaScript users of the Fetch API do their own manual decompression. Just flying over the docs the Response object seems to have a number of methods like arrayBuffer() blob() json() text() which sound like some of them would return decompressed data. Would it be an option to allow the client to specify which of these methods should be used? Though I'm assuming this is non-trivial in the context of Godot hiding any platform implementation details.

[Faless]

@lloesche the Fetch API (but also the old XHR API) does not give you control over those headers, but the browser will also automatically handle compression and decompression for you.
The engine should not do any compression or decompression in that case, which is, among other reasons, why gzip/deflate support has been reverted in 3.x (#53800).
The original PR (#38944) is still in master though, and should be either fixed or reverted before 4.0 stable.

[lloesche]

@Faless just to confirm, you're saying the browser is supposed to transparently decompress a compressed response? Because that's what I would have expected, but it's not what we saw in our testing yesterday. At the same time I don't want to rule out any snafu on our side.

Here's what we saw:
We have a Godot app that makes multiple requests to a REST API expecting JSON responses. The standalone build worked no problem. The HTML5 export only worked after we ensured server side that the Accept-Encoding headers got ignored and the server always returned an uncompressed response.

Again, I don't want to rule out that the issue was with something else, but when I used Wireshark to look at what the client (Godot app in Chrome) sent, the main difference between Standalone and HTML5 export I could see was that in the case of failing requests the browser had added an Accept-Encoding header

and the server response got compressed:

On the Godot side this resulted in:

When debug printing the response in the standalone build we saw JSON data, in the HTML5 build we saw random binary data, which we assumed was the compressed response (we did not confirm this though by actually trying to deflate decompress that binary data).

After we disabled compression server-side for one API route (the one our first request goes to), requests to that endpoint succeeded on the HTML5 build. When we disabled it for all routes all requests succeeded. So we just made the educated assumption that compression was the culprit. But again, not ruling out anything else as we also found and fixed a number of other issues during debugging, so focussing on the compression headers might have been a red herring.
If you're saying we should never see the compressed response in Godot then we'll have to dig back in and debug further.

Likely a good idea anyways as my concern with ignoring the headers in the backend (or hosting a proxy as suggested above) is that even when we turn off compression server side, there might still be content-proxies and load balancers in between the client and the server that will see the Accept-Encoding header and then try to be "helpful" by compressing the uncompressed server response.

[Faless]

If you're saying we should never see the compressed response in Godot then we'll have to dig back in and debug further.

That is the case in HTML5 builds, yes.

[lloesche]

If you're saying we should never see the compressed response in Godot then we'll have to dig back in and debug further.

That is the case in HTML5 builds, yes.

Okay, can confirm that's the case. We just retested our latest changes including the server side compression and it still works. So sorry for the noise above! We obviously messed up in our testing env yesterday somewhere.