Fix severe compression bug introduced by #97

[henry40408]

closes #98

How to reproduce

• Prepare a large file, e.g. Frankenstein; Or, The Modern Prometheus by Mary Wollstonecraft Shelley, whose size should be greater than 4KB. Save it as x.txt.
• Check out master 70b431a.
• cargo run
• curl --compressed http://localhost:5000/x.txt

A error like curl: (23) Failed reading the chunked-encoded stream will occur and only first 4KB content of file will be returned.

Root cause

sfz/src/server/serve.rs

Lines 416 to 419 in 70b431a

	.map(|b| match b {
	Ok(b) => compress(&b, content_encoding),
	Err(e) => Err(e),
	});

Maybe the body stream isn't split like how I expected, so only the first item of stream get compressed and returned then the rest are dropped. Since the buffer of file stream is 4KB, it might explain why only the first 4KB will be returned:

sfz/src/server/send.rs

Line 160 in 70b431a

let mut buf = BytesMut::zeroed(4_096);

Solution

• I replace compress with async-compression.
• Compared to Stream response body #97 (comment) , there is no significant performance improvement or loss when client enables deflate or gzip.
• When it comes to brotli, compression level and response time are highly relevant, so this pull request also resolves Improve brotli compression performance #98 .

brotli	level = 6	level = 0, fastest
benchmark	Running 10s test @ http://localhost:5000/Cargo.toml 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 5.45ms 2.20ms 23.10ms 72.66% Req/Sec 0.93k 79.59 1.06k 70.00% 18434 requests in 10.01s, 18.09MB read Requests/sec: 1841.70 Transfer/sec: 1.81MB	Running 10s test @ http://localhost:5000/Cargo.toml 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 2.97ms 1.09ms 19.27ms 69.77% Req/Sec 1.70k 110.24 3.07k 96.52% 33932 requests in 10.10s, 38.96MB read Requests/sec: 3359.58 Transfer/sec: 3.86MB
response size (1561)	700 (44%)	875 (56%)

• Another example: Pride and Prejudice by Jane Austen

brotli	level = 6	level = 0, fastest
benchmark	Running 10s test @ http://localhost:5000/1342-0.txt 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 1.30s 432.24ms 1.79s 76.92% Req/Sec 4.71 3.89 20.00 78.57% 68 requests in 10.01s, 16.04MB read Socket errors: connect 0, read 0, write 0, timeout 3 Requests/sec: 6.79 Transfer/sec: 1.60MB	Running 10s test @ http://localhost:5000/1342-0.txt 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 880.53ms 337.25ms 1.51s 68.52% Req/Sec 6.25 4.69 30.00 89.47% 108 requests in 10.01s, 42.28MB read Requests/sec: 10.78 Transfer/sec: 4.22MB
response size (798774)	246517 (30%)	405123 (50%)

[weihanglo]

Nice rework! I am fine with lowering brotli compression level.

Just some nits, nothing prominent so far. Thank you!

[weihanglo]

Thank you for getting this job done!