Slow download speed of HTTPRequest node

[neqs20]

Godot version:

• v3.1.1.stable.official

OS/device including version:

• Windows 10 x64 build: 17763.805

Issue description:

I wanted to download a file from my server ( Ubuntu Server 18.04 ) that is connected to my local network. The file has 100MB. As I found out, the best way to do this is over HTTPRequest node. The problem is with download speed. I have set up new project that prints the download speed (I included the zipped project). I have also done some tests. Here I included them:

Bandwidth: 200 Mb/s

First Test

File: From my local server.
File size: 100 MB

• Project code:

1. Download speed: ~2.2 MB/s
2. Time: ~48 seconds
3. Use threads: true

• Browser:

1. Download speed: ~40 MB/s
2. Time: ~2 seconds

Second Test

File: from https://speed.hetzner.de/
File size: 100 MB

• Project code:

1. Download speed: ~2.0 MB/s
2. Time: ~50 seconds
3. Use threads: true

• Browser:

1. Download speed: ~4.1 MB/s
2. Time: ~21 seconds

The difference is clearly visible.

Steps to reproduce:

• Create new script
• Copy and paste following code:

var done := false

var last_bytes := 0

var time := 0.0 # timer that is restarted ~every second

var http

func _ready():
	http = HTTPRequest.new()
	add_child(http)
	http.use_threads = true
	http.download_file = "100MB.bin"
	http.connect("request_completed", self, "_on_file_request_completed")

	# zip file (100 MB) from my local server 
	#http.request("http://192.168.100.104:4220/patch2.0.0.zip"

	# 100MB test file
	http.request("https://speed.hetzner.de/100MB.bin")


func _process(delta):
	if not done and time >= 1:
		var speed_in_bytes = http.get_downloaded_bytes() - last_bytes
		var speed_in_mega_bytes = speed_in_bytes / 1000000.0
		
		print(speed_in_mega_bytes, " MB/s")
		
		last_bytes = http.get_downloaded_bytes()
		time = 0
	time += delta

func _on_file_request_completed(result, response_code, headers, body):
	done = true
	print("File downloaded")

• Attach this script to any node
• Run it
• Check output

Minimal reproduction project:
To import this project use the import feature at the project manager
HttpTest.zip

[Mellondill]

Same for me

[Mellondill]

I've found some workaround, don't know if it would be suitable for you, but though here is the code that could help you:

func _process(delta):
	if resource_requested:
		var t = OS.get_ticks_msec()
		while OS.get_ticks_msec() < t + max_time_ms && resource_requested:
			$req.notification(NOTIFICATION_INTERNAL_PROCESS)

where:
$req - HTTPRequest Node
max_time_ms - max time of polling data per_frame (for my needs it was ok from 4 to 10 ms)

[neqs20]

Well, I've done some tests and it doesn't solve problem at all. I don't know what you meant Mellondill but the download speed didn't change. Could explain the code you made ?

[Mellondill]

Hello @HazmatDemon, I've tried to dive deep in classes HTTPRequest and HTTPClient, and found that actual polling of data from TCP socket is done in HTTPClient::poll() method.
HTTPClient::poll() method is called in HTTPRequest::_update_connection() method, and this method is called in HTTPRequest::_notification(int p_what) method.
Method HTTPRequest::_notification(int p_what) with argument NOTIFICATION_INTERNAL_PROCESS is called ones per frame.

Due to default read chunk size is 4096 bytes, and due to some circumstances(response headers don't contain 'content-length' attribute, etc.) that the left body length of response couldn't be determined, HTTPRequest just polls only 4096 bytes per frame.

I've just thought if I can make polling of data not per frame but in a controlled loop.

By 'controlled' I meant limiting of polling data in loop for about 5-10 milliseconds per frame, for your application to not have lags, and have stable FPS while data is polled.

So basically I'm speeding up HTTPRequest data polling, and it worked for me, and I thought that it could be suitable for you too!

[Calinou]

cc @Faless

What would be a good way to solve this? Would a custom timer work for this purpose?

[Faless]

What would be a good way to solve this? Would a custom timer work for this purpose?

I'm not sure, blocking for some milliseconds is not an option.

The repro project uses threads, and threads should poll continuously (keep blocking) so I'm not sure why this is happening.

[Faless]

I haven't had time to investigate this further sadly

[AlexHolly]

I am not able to reproduce on linux Mint
0ab0d11
Getting 5.9 MB/s in Godot and browser

On Windows
Getting 2.0 MB/s in Godot
5.5 MB/s in browser

[neqs20]

I tested it again (the same code from earlier), this time on Ubuntu and Manjaro and got 10 - 12 MB/s. It seems like Linux doesn't have problems at all.

[Faless]

I was able to reproduce this on windows, and it seems related to the small chunk size we were using in the HTTPClient. I have exposed the property to do change it in the above PR, but we should evaluate the possibility of a better default.

[vitorventurin]

Same issue here... any progress?

[Calinou]

@vitorventurin Did you increase the download chunk size in HTTPRequest? The default is still 4 KiB, which is slow for downloading large files. See #33862.