tune x264 threading

[totaam]

Issue migrated from trac ticket # 1840

component: encodings | priority: major | resolution: worksforme

2018-05-13 07:55:39: antoine created the issue

---

Somewhat related to #1839 since we now use vp8 / vp9 more than h264 in the python client.

References:

• Why sliced thread affect so much on realtime encoding using ffmpeg x264?: frame-based threading has better throughput than slice-based
• x264 doc threads: Each slice adds some bitrate (or equivalently reduces quality) ... Some parts of the encoder are serial, so it doesn't scale well with lots of cpus.

[totaam]

2018-05-13 08:00:03: antoine changed owner from antoine to maxmylyn

[totaam]

2018-05-13 08:00:03: antoine edited the issue description

[totaam]

2018-05-13 08:00:03: antoine commented

---

r19300 disables sliced threads when speed is below XPRA_X264_SLICED_THREADS (which defaults to 60).

@maxmylyn: please use the automated tests to see what impact this has on h264 encoding.
In particular, CPU load, latency, bandwidth, etc.
Because sliced based threading is so inefficient, you may want to sacrifice a little bit of latency to get better user density.

See also #1851 for vp8 / vp9.

[totaam]

2018-05-21 19:27:18: maxmylyn commented

---

Okay I will be investigating the automated test box this week. I have a few issues I have to tackle.

Still TODO:

• Update the box from Fedora 26 -> Fedora 27 since 26 is EOL soon(already?)

• Fix the virtual display resolution -> currently stuck on a super low res display (likely because it's plugged in to a VGA KVM switch)

• Figure out why the IPTable rules aren't loading on boot -> No packet accounting

[totaam]

2018-05-29 18:50:49: maxmylyn commented

---

Okay the box is updated to Fedora 28 (took much longer than it really should have). The packet accounting is also working again. However it's still not auto-loading the IPTables configuration - I'll have to give yet another different method of loading the IPTables config a try.

I'm going to write a quick bash script to check the H264 performance and also the VPX performance for #1851 and run that in the background today.

I'm also going to write a Python (it occurs to me I could write a Bash script, too) helper file that should make these one-off test runs easier to run. As is I have to write a custom Bash script and run it via nohup since the test box is about a 45 minute drive away and the VPN connection is unstable at best.

[totaam]

2018-11-27 09:04:02: antoine commented

---

See #2064#comment:3

[totaam]

2019-01-03 18:03:40: maxmylyn commented

---

My mathematician is unavailable for a week or so, so I'm going to post the output I've got so far. No charts or anything, just the raw CSVs and logs.

[totaam]

2019-01-03 18:04:38: maxmylyn uploaded file h264.zip (108.8 KiB)

H264 test output data organized by XPRA_X264_SLICED_THREADS value

[totaam]

2019-01-18 11:32:43: antoine commented

---

Most of the data is missing from those data files (batch delay, quality, etc), so there is no way of knowing if sliced threads were actually used or not. (that's probably something caused by the xpra info changes in 2.5 - the tests code will need updating if that's the case)

Testing with XPRA_X264_SLICED_THREADS=60 does not do anything since that's the default value. Use 1 to always enable, and 100 to disable.

Also, we want to test the changes in the x264 video encoding settings, so there's not much point in testing "xterm" or "gtkperf" as those aren't meant to trigger any video at all. The only thing these could be useful for would be to see the variance between test runs, but you might as well get that from multiple runs with video.

[totaam]

2019-02-09 03:42:13: antoine changed owner from maxmylyn to encodedEntropy

[totaam]

2019-08-01 12:58:47: smo changed owner from encodedEntropy to smo

[totaam]

2019-08-02 16:22:10: smo uploaded file testx264chart.tar.gz (121.3 KiB)

[totaam]

2019-08-02 16:24:35: smo changed owner from smo to Antoine Martin

[totaam]

2019-08-02 16:24:35: smo commented

---

Attached some generated charts with XPRA_X264_SLICED_THREADS set to 1 and 100.

All the csv from the 2 reps and logs are there if you like in the archive.

The cpu changes don't look that huge to me but take a look for yourself.

[totaam]

2019-08-03 16:19:56: antoine changed status from new to closed

[totaam]

2019-08-03 16:19:56: antoine set resolution to worksforme

[totaam]

2019-08-03 16:19:56: antoine commented

---

Most metrics look better when sliced threads are enabled more aggressively (XPRA_X264_SLICED_THREADS=1): the batch delay is lower, pixels sent and regions/s higher.
Even the slightly greater bandwidth usage can be explained by the higher framerate.
But since there's not much difference, let's just leave the default value as it is.
Closing at last!