Compress/Encrypt Blocks in background

[jarifibrahim]

This PR adds support for block compression/encryptions in the background.

The following code was used for benchmarking (builder.go BenchmarkBuilder)

    bench := func(b *testing.B, opt *Options) {    
        // KeyCount * (keySize + ValSize)    
        b.SetBytes(int64(keysCount) * (32 + 32))    
        opt.BlockSize = 4 * 1024    
        opt.BloomFalsePositive = 0.01    
    
        //opt.TableSize = 5 << 20    
        b.ResetTimer()    
    
        for i := 0; i < b.N; i++ {    
            builder := NewTableBuilder(*opt)    
            for j := 0; j < keysCount; j++ {    
                builder.Add(keyList[j], vs, 0)    
            }    
            _ = builder.Finish()    
        }    
    }    
    
    b.Run("no compression", func(b *testing.B) {    
        var opt Options    
        opt.Compression = options.None    
        bench(b, &opt)    
    })    
    b.Run("encryption", func(b *testing.B) {    
        var opt Options    
        key := make([]byte, 32)    
        rand.Read(key)    
        opt.DataKey = &pb.DataKey{Data: key}    
        bench(b, &opt)    
    })    
    b.Run("zstd compression", func(b *testing.B) {    
        var opt Options    
        opt.Compression = options.ZSTD    
        b.Run("level 1", func(b *testing.B) {    
            opt.ZSTDCompressionLevel = 1    
            bench(b, &opt)
        })
        b.Run("level 3", func(b *testing.B) {
            opt.ZSTDCompressionLevel = 3
            bench(b, &opt)
        })
        b.Run("level 15", func(b *testing.B) {
            opt.ZSTDCompressionLevel = 15
            bench(b, &opt)
        })
    })

Master branch vs This Branch (new) Benchmarks.

benchstat master.txt new.txt
name                                   old time/op    new time/op     delta (-ve is better. Reduction in time)
Builder1/no_compression-16                177ms ± 1%      177ms ± 2%      ~     (p=0.690 n=5+5)
Builder1/encryption-16                    313ms ± 4%      245ms ± 3%   -21.69%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_1-16      330ms ± 1%      227ms ± 2%   -31.22%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_3-16      345ms ± 2%      227ms ± 3%   -34.39%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_15-16     10.7s ± 1%       1.2s ± 0%   -88.96%  (p=0.008 n=5+5)

name                                   old speed      new speed       delta (+ve is better. Speed improvement)
Builder1/no_compression-16              471MB/s ± 1%    471MB/s ± 2%      ~     (p=0.690 n=5+5)
Builder1/encryption-16                  266MB/s ± 4%    340MB/s ± 2%   +27.67%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_1-16    252MB/s ± 1%    367MB/s ± 2%   +45.40%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_3-16    241MB/s ± 2%    367MB/s ± 3%   +52.44%  (p=0.008 n=5+5)
Builder1/zstd_compression/level_15-16  7.80MB/s ± 1%  70.62MB/s ± 0%  +805.33%  (p=0.008 n=5+5)

---

This change is 

[jarifibrahim]

Reviewable status: 0 of 8 files reviewed, 2 unresolved discussions (waiting on @ashish-goswami and @manishrjain)

---

db2_test.go, line 678 at r1 (raw file):

		db1.valueDirGuard = nil
	}
	require.NoError(t, db1.Close())

This is a small cleanup which is not related to this PR.

---

table/builder.go, line 359 at r1 (raw file):

		start := uint32(0)
		for i, bl := range b.blockList {
			// Length of the block is start minues the end.

minues => minus.

[coveralls]

Coverage increased (+0.2%) to 69.944% when pulling c5e3d30 on ibrahim/replace-b.buf into bce069c on master.

[manishrjain]

Is this ready for review? @jarifibrahim

Reviewable status: 0 of 8 files reviewed, 2 unresolved discussions (waiting on @ashish-goswami and @manishrjain)