Skip to content

Commit

Permalink
Compress/Encrypt Blocks in the background (#1227)
Browse files Browse the repository at this point in the history
This commit adds support for compression/encryption in the background.
All incoming writes are saved to a common buffer and then worker go
routine pick up each block (a block is just a start and end point in
the main buffer). The worker goroutines also fix the table start and
end points. We fix the interleaving space (which might be caused because
the block was compressed) in the builder.Finish() call. All the blocks
are copied over to their correct position.

Master branch vs This commit (new) Benchmarks.
```
benchstat master.txt new.txt
name                              old time/op    new time/op  delta (-ve is better. Reduction in time)
no_compression-16                177ms ± 1%      177ms ± 2%      ~     (p=0.690 n=5+5)
encryption-16                    313ms ± 4%      245ms ± 3%   -21.69%  (p=0.008 n=5+5)
zstd_compression/level_1-16      330ms ± 1%      227ms ± 2%   -31.22%  (p=0.008 n=5+5)
zstd_compression/level_3-16      345ms ± 2%      227ms ± 3%   -34.39%  (p=0.008 n=5+5)
zstd_compression/level_15-16     10.7s ± 1%       1.2s ± 0%   -88.96%  (p=0.008 n=5+5)

name                           old speed       new speed       delta (+ve is better. Speed improvement)
no_compression-16              471MB/s ± 1%    471MB/s ± 2%      ~     (p=0.690 n=5+5)
encryption-16                  266MB/s ± 4%    340MB/s ± 2%   +27.67%  (p=0.008 n=5+5)
zstd_compression/level_1-16    252MB/s ± 1%    367MB/s ± 2%   +45.40%  (p=0.008 n=5+5)
zstd_compression/level_3-16    241MB/s ± 2%    367MB/s ± 3%   +52.44%  (p=0.008 n=5+5)
zstd_compression/level_15-16  7.80MB/s ± 1%  70.62MB/s ± 0%  +805.33%  (p=0.008 n=5+5)
``` 

Please look at #1227 (comment)
for the code used for benchmarking.
  • Loading branch information
Ibrahim Jarif authored Mar 12, 2020
1 parent bdb2b13 commit b13b927
Show file tree
Hide file tree
Showing 8 changed files with 293 additions and 116 deletions.
9 changes: 3 additions & 6 deletions db2_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -669,16 +669,13 @@ func TestL0GCBug(t *testing.T) {
// Simulate a crash by not closing db1 but releasing the locks.
if db1.dirLockGuard != nil {
require.NoError(t, db1.dirLockGuard.release())
db1.dirLockGuard = nil
}
if db1.valueDirGuard != nil {
require.NoError(t, db1.valueDirGuard.release())
db1.valueDirGuard = nil
}
for _, f := range db1.vlog.filesMap {
require.NoError(t, f.fd.Close())
}
require.NoError(t, db1.registry.Close())
require.NoError(t, db1.lc.close())
require.NoError(t, db1.manifest.close())
require.NoError(t, db1.Close())

db2, err := Open(opts)
require.NoError(t, err)
Expand Down
1 change: 1 addition & 0 deletions options.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ func DefaultOptions(path string) Options {

func buildTableOptions(opt Options) table.Options {
return table.Options{
TableSize: uint64(opt.MaxTableSize),
BlockSize: opt.BlockSize,
BloomFalsePositive: opt.BloomFalsePositive,
LoadingMode: opt.TableLoadingMode,
Expand Down
Loading

0 comments on commit b13b927

Please sign in to comment.