Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flate: Fix/tweak huffman bit estimates #206

Merged
merged 9 commits into from
Jan 18, 2020

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Jan 14, 2020

New blocks where given an advantage since estimates for new blocks did not include extra bits.

Optimize log2 calculation with faster estimate:

Benchmark_tokens_EstimatedBits-32    	  299931	      4062 ns/op	   0.25 MB/s
Benchmark_tokens_EstimatedBits-32    	 2201341	       543 ns/op	   1.84 MB/s

Before/After

file	out	level	insize	outsize	millis	mb/s
gob-stream	gzkp	-2	1911399616	1410346540	6927	263.13
gob-stream	gzkp	1	1911399616	360099905	5708	319.33
gob-stream	gzkp	2	1911399616	346448934	6334	287.77
gob-stream	gzkp	3	1911399616	339896850	8308	219.39
gob-stream	gzkp	4	1911399616	319850581	9092	200.49
gob-stream	gzkp	5	1911399616	310556560	10675	170.75
gob-stream	gzkp	6	1911399616	309710534	12074	150.96
gob-stream	gzkp	7	1911399616	305960448	14837	122.86
gob-stream	gzkp	8	1911399616	300291259	17893	101.88
gob-stream	gzkp	9	1911399616	280306976	31682	57.54
file	out	level	insize	outsize	millis	mb/s
gob-stream	gzkp	-2	1911399616	1410346519	6767	269.35
gob-stream	gzkp	1	1911399616	361916183	5635	323.47
gob-stream	gzkp	2	1911399616	349857310	6256	291.36
gob-stream	gzkp	3	1911399616	341885261	6885	264.74
gob-stream	gzkp	4	1911399616	321792143	7890	231.01
gob-stream	gzkp	5	1911399616	312085491	8911	204.54
gob-stream	gzkp	6	1911399616	311292135	11114	164.01
gob-stream	gzkp	7	1911399616	305960448	14712	123.90
gob-stream	gzkp	8	1911399616	300291259	17781	102.51
gob-stream	gzkp	9	1911399616	280306976	31161	58.50

file	out	level	insize	outsize	millis	mb/s
github-june-2days-2019.json	gzkp	-2	6273951764	4191425185	22107	270.64
github-june-2days-2019.json	gzkp	1	6273951764	1150816630	18647	320.87
github-june-2days-2019.json	gzkp	2	6273951764	1112997952	21263	281.39
github-june-2days-2019.json	gzkp	3	6273951764	1103339187	25701	232.80
github-june-2days-2019.json	gzkp	4	6273951764	997880134	28780	207.89
github-june-2days-2019.json	gzkp	5	6273951764	963159098	34158	175.16
github-june-2days-2019.json	gzkp	6	6273951764	949542385	39269	152.36
github-june-2days-2019.json	gzkp	7	6273951764	934872037	49069	121.94
github-june-2days-2019.json	gzkp	8	6273951764	921116719	57011	104.95
github-june-2days-2019.json	gzkp	9	6273951764	899793019	84347	70.94
file	out	level	insize	outsize	millis	mb/s
github-june-2days-2019.json	gzkp	-2	6273951764	4196148824	21661	276.22
github-june-2days-2019.json	gzkp	1	6273951764	1125991387	18940	315.90
github-june-2days-2019.json	gzkp	2	6273951764	1107583034	21865	273.64
github-june-2days-2019.json	gzkp	3	6273951764	1103676413	23369	256.03
github-june-2days-2019.json	gzkp	4	6273951764	999153827	25717	232.65
github-june-2days-2019.json	gzkp	5	6273951764	964959169	29933	199.89
github-june-2days-2019.json	gzkp	6	6273951764	951460252	35396	169.03
github-june-2days-2019.json	gzkp	7	6273951764	934872037	48143	124.28
github-june-2days-2019.json	gzkp	8	6273951764	921116719	56076	106.70
github-june-2days-2019.json	gzkp	9	6273951764	899793019	83476	71.68

file	out	level	insize	outsize	millis	mb/s
enwik9	gzkp	-2	1000000000	651299040	3615	263.75
enwik9	gzkp	1	1000000000	384456678	6512	146.44
enwik9	gzkp	2	1000000000	372262266	6973	136.76
enwik9	gzkp	3	1000000000	364294282	8140	117.15
enwik9	gzkp	4	1000000000	341904112	9049	105.39
enwik9	gzkp	5	1000000000	338375880	10652	89.52
enwik9	gzkp	6	1000000000	336071599	11201	85.14
enwik9	gzkp	7	1000000000	329248670	18506	51.53
enwik9	gzkp	8	1000000000	326623117	23616	40.38
enwik9	gzkp	9	1000000000	324205510	29803	32.00
file	out	level	insize	outsize	millis	mb/s
enwik9	gzkp	-2	1000000000	652568269	3534	269.85
enwik9	gzkp	1	1000000000	383140821	6523	146.19
enwik9	gzkp	2	1000000000	372365357	7062	135.04
enwik9	gzkp	3	1000000000	364568604	7674	124.26
enwik9	gzkp	4	1000000000	342411924	8418	113.28
enwik9	gzkp	5	1000000000	338866672	9879	96.53
enwik9	gzkp	6	1000000000	336511427	10474	91.05
enwik9	gzkp	7	1000000000	329248670	18497	51.56
enwik9	gzkp	8	1000000000	326623117	24016	39.71
enwik9	gzkp	9	1000000000	324205510	29511	32.32

file	out	level	insize	outsize	millis	mb/s
10gb.tar	gzkp	-2	10065157632	6540870519	30341	316.36
10gb.tar	gzkp	1	10065157632	5280659104	45780	209.67
10gb.tar	gzkp	2	10065157632	5147972222	48354	198.51
10gb.tar	gzkp	3	10065157632	5059091941	54003	177.75
10gb.tar	gzkp	4	10065157632	4947136306	59389	161.63
10gb.tar	gzkp	5	10065157632	4933679787	71788	133.71
10gb.tar	gzkp	6	10065157632	4895606626	76606	125.30
file	out	level	insize	outsize	millis	mb/s
10gb.tar	gzkp	-2	10065157632	6548123344	29288	327.73
10gb.tar	gzkp	1	10065157632	5214106110	45982	208.75
10gb.tar	gzkp	2	10065157632	5120120494	48900	196.29
10gb.tar	gzkp	3	10065157632	5047291855	52904	181.44
10gb.tar	gzkp	4	10065157632	4943352328	55613	172.60
10gb.tar	gzkp	5	10065157632	4931789557	67581	142.03
10gb.tar	gzkp	6	10065157632	4894858258	71080	135.04
10gb.tar	gzkp	7	10065157632	4849130270	204530	46.93
name                        old speed      new speed      delta
EncodeDigitsConstant1e4-12   349MB/s ± 1%   355MB/s ± 0%    ~     (p=0.107 n=3+3)
EncodeDigitsConstant1e5-12   326MB/s ± 0%   333MB/s ± 1%  +2.24%  (p=0.008 n=3+3)
EncodeDigitsConstant1e6-12   368MB/s ± 1%   381MB/s ± 0%  +3.58%  (p=0.008 n=3+3)
EncodeDigitsSpeed1e4-12      132MB/s ± 1%   135MB/s ± 1%  +2.35%  (p=0.024 n=3+3)
EncodeDigitsSpeed1e5-12      103MB/s ± 0%   102MB/s ± 0%    ~     (p=0.276 n=3+3)
EncodeDigitsSpeed1e6-12      103MB/s ± 0%   107MB/s ± 0%  +4.10%  (p=0.001 n=3+3)
EncodeDigitsDefault1e4-12   65.6MB/s ± 0%  66.7MB/s ± 1%    ~     (p=0.060 n=3+3)
EncodeDigitsDefault1e5-12   60.4MB/s ± 1%  61.7MB/s ± 1%  +2.02%  (p=0.018 n=3+3)
EncodeDigitsDefault1e6-12   59.2MB/s ± 0%  60.1MB/s ± 1%    ~     (p=0.103 n=3+3)
EncodeDigitsCompress1e4-12  45.8MB/s ± 0%  45.8MB/s ± 0%    ~     (p=0.552 n=3+3)
EncodeDigitsCompress1e5-12  26.5MB/s ± 1%  26.7MB/s ± 0%    ~     (p=0.354 n=3+3)
EncodeDigitsCompress1e6-12  24.6MB/s ± 1%  24.9MB/s ± 0%  +1.23%  (p=0.041 n=3+3)
EncodeDigitsSL1e4-12        76.4MB/s ± 0%  78.6MB/s ± 0%  +2.83%  (p=0.000 n=3+3)
EncodeDigitsSL1e5-12         112MB/s ± 0%   117MB/s ± 0%  +5.21%  (p=0.001 n=3+3)
EncodeDigitsSL1e6-12         120MB/s ± 0%   126MB/s ± 1%  +5.07%  (p=0.000 n=3+3)
EncodeTwainConstant1e4-12    255MB/s ± 1%   278MB/s ± 0%  +9.29%  (p=0.002 n=3+3)
EncodeTwainConstant1e5-12    289MB/s ± 1%   301MB/s ± 0%  +4.24%  (p=0.014 n=3+3)
EncodeTwainConstant1e6-12    339MB/s ± 1%   347MB/s ± 0%    ~     (p=0.055 n=3+3)
EncodeTwainSpeed1e4-12      98.5MB/s ± 0%  98.8MB/s ± 0%    ~     (p=0.108 n=3+3)
EncodeTwainSpeed1e5-12       104MB/s ± 1%   105MB/s ± 0%    ~     (p=0.241 n=3+3)
EncodeTwainSpeed1e6-12       112MB/s ± 0%   113MB/s ± 0%  +1.03%  (p=0.019 n=3+3)
EncodeTwainDefault1e4-12    65.9MB/s ± 1%  65.3MB/s ± 1%    ~     (p=0.092 n=3+3)
EncodeTwainDefault1e5-12    68.5MB/s ± 0%  68.7MB/s ± 0%    ~     (p=0.114 n=3+3)
EncodeTwainDefault1e6-12    67.1MB/s ± 2%  70.8MB/s ± 3%  +5.46%  (p=0.049 n=3+3)
EncodeTwainCompress1e4-12   37.3MB/s ± 0%  37.5MB/s ± 1%    ~     (p=0.297 n=3+3)
EncodeTwainCompress1e5-12   20.9MB/s ± 1%  20.6MB/s ± 2%    ~     (p=0.197 n=3+3)
EncodeTwainCompress1e6-12   19.0MB/s ± 0%  18.6MB/s ± 1%  -1.90%  (p=0.018 n=3+3)
EncodeTwainSL1e4-12         70.8MB/s ± 0%  69.9MB/s ± 0%  -1.14%  (p=0.012 n=3+3)
EncodeTwainSL1e5-12          111MB/s ± 0%   111MB/s ± 2%    ~     (p=0.635 n=3+3)
EncodeTwainSL1e6-12          120MB/s ± 1%   123MB/s ± 1%  +2.26%  (p=0.037 n=3+3)

Compression size changes pending...

New blocks where given an advantage since estimates for new blocks did not include extra bits.

Optimize log2 calculation with faster estimate:

```
Benchmark_tokens_EstimatedBits-32    	  299931	      4062 ns/op	   0.25 MB/s
Benchmark_tokens_EstimatedBits-32    	 2201341	       543 ns/op	   1.84 MB/s
```
@klauspost klauspost changed the title Flate fix huffman bit estimates Flate: Fix/tweak huffman bit estimates Jan 16, 2020
@klauspost klauspost merged commit 1cf5cb2 into master Jan 18, 2020
@klauspost klauspost deleted the flate-fix-huffman-bit-estimates branch January 18, 2020 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant