sstable: reduce writer allocations during index flushing

[dt]

While working on a separate change that included passing a large number
of pre-generated blocks all to addIndexEntry in a tight loop, some of
the allocations it was doing per block flushed became much more apparent in
profiles. By batch allocating keys used as separators, and by adjusting
the buffering of filled sub-index blocks to store their finished byte
representation (also batch allocated) with relevant details, rather than
the whole block writer, that blockWriter can instead be reused, which
importantly allows reuse of its allocated buffers eliminating a significant source of
allocations.

There overall performance impact to Writer is negligible but was more pronounced
in another change that flushes blocks in a tight loop.

name                                                          old time/op    new time/op    delta
Writer/block=4.0_K/filter=false/compression=NoCompression-10    50.7ms ± 0%    51.5ms ± 4%      ~     (p=0.792 n=5+6)
Writer/block=4.0_K/filter=false/compression=Snappy-10           62.8ms ± 2%    62.5ms ± 3%      ~     (p=0.310 n=6+6)
Writer/block=32_K/filter=false/compression=NoCompression-10     48.9ms ± 1%    48.8ms ± 1%      ~     (p=1.000 n=6+6)
Writer/block=32_K/filter=false/compression=Snappy-10            60.3ms ± 4%    59.0ms ± 5%      ~     (p=0.310 n=6+6)

name                                                          old speed      new speed      delta
Writer/block=4.0_K/filter=false/compression=NoCompression-10   756MB/s ± 0%   744MB/s ± 4%      ~     (p=0.792 n=5+6)
Writer/block=4.0_K/filter=false/compression=Snappy-10          156MB/s ± 2%   156MB/s ± 3%      ~     (p=0.310 n=6+6)
Writer/block=32_K/filter=false/compression=NoCompression-10    773MB/s ± 1%   774MB/s ± 1%      ~     (p=1.000 n=6+6)
Writer/block=32_K/filter=false/compression=Snappy-10           113MB/s ± 4%   115MB/s ± 4%      ~     (p=0.310 n=6+6)

name                                                          old alloc/op   new alloc/op   delta
Writer/block=4.0_K/filter=false/compression=NoCompression-10    1.18MB ± 0%    0.80MB ± 0%   -32.14%  (p=0.002 n=6+6)
Writer/block=4.0_K/filter=false/compression=Snappy-10           1.20MB ± 0%    0.83MB ± 0%   -31.08%  (p=0.002 n=6+6)
Writer/block=32_K/filter=false/compression=NoCompression-10      252kB ± 0%     952kB ± 0%  +278.61%  (p=0.002 n=6+6)
Writer/block=32_K/filter=false/compression=Snappy-10             456kB ± 0%    1157kB ± 0%  +153.55%  (p=0.002 n=6+6)

name                                                          old allocs/op  new allocs/op  delta
Writer/block=4.0_K/filter=false/compression=NoCompression-10     10.7k ± 0%      0.1k ± 0%   -99.03%  (p=0.002 n=6+6)
Writer/block=4.0_K/filter=false/compression=Snappy-10            10.7k ± 0%      0.1k ± 0%   -98.97%  (p=0.002 n=6+6)
Writer/block=32_K/filter=false/compression=NoCompression-10      1.27k ± 0%     0.10k ± 0%   -92.20%  (p=0.002 n=6+6)
Writer/block=32_K/filter=false/compression=Snappy-10             1.27k ± 0%     0.10k ± 0%   -91.84%  (p=0.002 n=6+6)

[cockroach-teamcity]

This change is 

[jbowens]

Reviewed 1 of 1 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dt)

---

sstable/writer.go, line 536 at r2 (raw file):

			w.indexBlockAlloc = make([]byte, len(bk)*16)
		} else {
			w.indexBlockAlloc = make([]byte, len(bk)*32)

how deliberate or empirical is the choice of these constants and the splitting of these two cases? wonder if it's worth the special casing, or if it is, we should do something adaptive (eg, 2 × the size of our previous indexBlockAlloc allocation)

[dt]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @jbowens)

---

sstable/writer.go, line 536 at r2 (raw file):

Previously, jbowens (Jackson Owens) wrote…

how deliberate or empirical is the choice of these constants and the splitting of these two cases? wonder if it's worth the special casing, or if it is, we should do something adaptive (eg, 2 × the size of our previous indexBlockAlloc allocation)

Oh, I'm not actually sure I meant to leave the two cases here. I might just make it a constant 16 for now?

[jbowens]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dt)

---

sstable/writer.go, line 536 at r2 (raw file):

Previously, dt (David Taylor) wrote…

Oh, I'm not actually sure I meant to leave the two cases here. I might just make it a constant 16 for now?

sgtm

[dt]

TFTR!