storage: fork-lift proposer-evaluated KV #10327
Conversation
tbg
added
proposer-evaluated-kv
S-1-stability
|
Did a first pass and this wasn't as bad as I would have expected. Still need to do a more detailed review. Review status: 0 of 13 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed. pkg/storage/replica.go, line 1762 at r1 (raw file):
Are you setting this just so pkg/storage/replica.go, line 1795 at r1 (raw file):
Comments from Reviewable |
|
Yeah, not horrible (I like to think the previous work has paid off). Definitely needs a few more iterations on my part, though; I'll do benchmarking to avoid performance regressions tomorrow and fill in more commentary. Review status: 0 of 13 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed. pkg/storage/replica.go, line 1795 at r1 (raw file):
|
|
Review status: 0 of 13 files reviewed at latest revision, 2 unresolved discussions, all commit checks successful. pkg/storage/replica.go, line 1762 at r1 (raw file):
|
|
Added benchmarks (commit message, copied into PR description). TL;DR is that with proposer-evaluated KV disabled, this PR mostly keeps the current performance profile. With proposer-eval'ed KV enabled, it's a bit worse but that makes sense since we're doing extra work serializing and deserializing each |
|
Looks good. Reviewed 7 of 13 files at r1, 4 of 6 files at r2, 2 of 2 files at r3. pkg/storage/replica_test.go, line 721 at r3 (raw file):
Put this in pkg/storage/storagebase/proposer_kv.proto, line 51 at r3 (raw file):
Any particular reason for the wrapper message instead of using a ChangeReplicasTrigger directly? pkg/storage/storagebase/proposer_kv.proto, line 131 at r3 (raw file):
Why a nested message? This might incur an extra copy of the data (I'm not sure). If you're using the optional nested message so you can distinguish absent and empty values, it may be better to use a separate bool. Comments from Reviewable |
tbg
force-pushed
the
exp-prop-kv
branch
2 times, most recently
from
2582d19 to
ac2e4a3
Compare
|
Review status: 7 of 17 files reviewed at latest revision, 5 unresolved discussions. pkg/storage/replica_test.go, line 721 at r3 (raw file):
|
|
I added CI, which roughly doubles test time. That isn't great, but I see no good way of preventing regressions (the first one was already introduced and fixed) especially since there is a fair number of PRs touching Review status: 7 of 17 files reviewed at latest revision, 5 unresolved discussions, some commit checks pending. Comments from Reviewable |
tbg
force-pushed
the
exp-prop-kv
branch
2 times, most recently
from
5a33f23 to
4df8ad3
Compare
|
To get the tests to pass (turns out some of them catch the lack of correctness in the command queue, albeit only via corruption during splits), I added a crude hack to address the problem in #10084 (adding an "everything" span during proposer-evaluated KV). |
|
Talked to @petermattis about testing: I'll add nightlies for this (i.e. don't review the changes in the teamcity scripts just yet; they're here only so I can be convinced CI passes reliably). |
|
Green except for some data races which I'm addressing now. Nothing that's really new; just our shoddy previous code being exposed in new ways. |
This fixes a source of data races in the experimental proposer-evaluated KV PR cockroachdb#10327. Updated `TestOptimizePuts` so that prior to the fix, it failed with ``` --- FAIL: TestOptimizePuts (0.01s) replica_test.go:1522: 2: optimizePuts mutated the original request slice: [[0].Put.Blind: false != true [1].Put.Blind: false != true [2].Put.Blind: false != true [3].Put.Blind: false != true [4].Put.Blind: false != true [5].Put.Blind: false != true [6].Put.Blind: false != true [7].Put.Blind: false != true [8].Put.Blind: false != true [9].Put.Blind: false != true] ```
With proposer-evaluated Raft, doing so would mutate the caller's transaction proto, with unclear (but definitely bad) implications. Do not mutate ba.Requests in optimizePuts. This fixes a source of data races in the experimental proposer-evaluated KV PR cockroachdb#10327. Updated `TestOptimizePuts` so that prior to the fix, it failed with ``` --- FAIL: TestOptimizePuts (0.01s) replica_test.go:1522: 2: optimizePuts mutated the original request slice: [[0].Put.Blind: false != true [1].Put.Blind: false != true [2].Put.Blind: false != true [3].Put.Blind: false != true [4].Put.Blind: false != true [5].Put.Blind: false != true [6].Put.Blind: false != true [7].Put.Blind: false != true [8].Put.Blind: false != true [9].Put.Blind: false != true] ```
|
Review status: 12 of 19 files reviewed at latest revision, 11 unresolved discussions, some commit checks pending. pkg/storage/replica.go, line 1958 at r5 (raw file):
|
|
Review status: 12 of 19 files reviewed at latest revision, 8 unresolved discussions, some commit checks pending. pkg/storage/replica.go, line 1958 at r5 (raw file):
|
|
Review status: 12 of 19 files reviewed at latest revision, 8 unresolved discussions, some commit checks pending. pkg/storage/replica.go, line 1958 at r5 (raw file):
|
|
Review status: 12 of 19 files reviewed at latest revision, 8 unresolved discussions, some commit checks pending. pkg/storage/replica.go, line 1958 at r5 (raw file):
|
|
Reviewed 1 of 13 files at r1, 2 of 15 files at r4, 6 of 7 files at r5, 1 of 1 files at r6. pkg/kv/txn_correctness_test.go, line 753 at r6 (raw file):
i don't understand these changes. why were they necessary? why can't you report all the errors? seems like a half-baked attempt to make this output more informative. pkg/storage/client_raft_test.go, line 950 at r6 (raw file):
you removed this "boom" but there's still a pkg/storage/client_split_test.go, line 1086 at r6 (raw file):
nit: propEvalKV is not a thing in this scope. pkg/storage/replica.go, line 1227 at r6 (raw file):
pkg/storage/replica.go, line 1796 at r6 (raw file):
no specific comment here but this is a reflection of @bdarnell's comment on the proto about these wrapper types. this basically forces a random allocation for unclear benefit. pkg/storage/storagebase/proposer_kv.go, line 19 at r4 (raw file):
|
|
Review status: 18 of 19 files reviewed at latest revision, 11 unresolved discussions, some commit checks failed. pkg/kv/txn_correctness_test.go, line 753 at r6 (raw file):
|
|
Reviewed 5 of 5 files at r7. pkg/kv/txn_correctness_test.go, line 753 at r6 (raw file):
|
|
Review status: all files reviewed at latest revision, 7 unresolved discussions, some commit checks failed. pkg/storage/client_raft_test.go, line 950 at r6 (raw file):
|
|
Reviewed 1 of 1 files at r8. pkg/storage/store_test.go, line 1392 at r7 (raw file):
|
|
Review status: all files reviewed at latest revision, 6 unresolved discussions, some commit checks failed. pkg/storage/store_test.go, line 1392 at r7 (raw file):
|
|
Reviewed 4 of 4 files at r9. Comments from Reviewable |
Fixes #6290. Add experimental proposer-evaluated KV gated behind the environment variable `COCKROACH_PROPOSER_EVALUATED_KV`. When set to a truthy value, Raft proposals are evaluated and the resulting RocksDB `WriteBatch` submitted to Raft along with some auxiliary metadata. The result of the evaluation is only stored in the pending command on the proposer, and returned to the waiting client after the `WriteBatch` has been applied. Introduce a natural failfast path for (most) proposals returning an error. Instead of proposing, waiting for Raft, and only then receiving an error, proposals which do not lead to a state change receive their error directly when the proposal is evaluated, upstream of Raft. Only errors which still want to persist data (for instance, `*TransactionRetryError` when intents were laid down) go through the whole proposal, with the client receiving the error after the associated `Batch` commits. While proposer-evaluated KV is now ready for review, preliminary testing and benchmarking, the current implementation is incomplete and incorrect: - `Lease` acquisition is not special-cased, meaning that lease state may be clobbered freely when non-leaseholders propose a lease request based on stale data. This needs to be fixed but it also shows that we don't stress that scenario sufficiently in testing yet. - Similarly, we don't check that commands can only apply under the lease that they were proposed (which is necessary). - `CommandQueue` does not account for internal keys accessed by overlapping commands correctly (this is tracked in #10084), which in principle also lead to anomalies which should be exposed by testing and addressed. Instead, **every** command inserts a span that covers everything. Horrible for performance, great for correctness; #10084 needs to address this. - `TestingCommandFilter` needs to be refactored to be an explicit interceptor for the pre-Raft stage of commands. Tests were fixed up enough to pass with proposer-evaluated KV as well, but it's possible that some tests don't test what they used to. Benchmark results for non-proposer-evaluated-KV against `master` below. **Note that all of them have the "correctness hack" in `(*Replica).beginCmds` disabled; not doing that should give much worse results but we don't care about those in practice** We pay a small fee in allocations, but nothing appears outrageous. So from a performance perspective, this PR seems safe to merge: ``` $ for br in exp-prop-kv masterdo git checkout $br && make bench COCKROACH_PROPOSER_EVALUATED_KV=false PKG=./sql TESTS='(Select|Insert|Update|Delete)1[0-9]*_Cockroach' TESTFLAGS='-benchmem -count 10' BENCHTIMEOUT=1h > $(git rev-parse --abbrev-ref HEAD); done $ benchstat master exp-prop-kv name old time/op new time/op delta Select1_Cockroach-24 79.8µs ± 2% 79.7µs ± 3% ~ (p=0.912 n=10+10) Insert1_Cockroach-24 500µs ± 3% 520µs ± 5% +4.12% (p=0.001 n=10+10) Insert10_Cockroach-24 665µs ± 3% 667µs ± 3% ~ (p=0.631 n=10+10) Insert100_Cockroach-24 1.77ms ± 5% 1.78ms ± 4% ~ (p=0.529 n=10+10) Insert1000_Cockroach-24 11.9ms ± 3% 11.9ms ± 2% ~ (p=1.000 n=10+10) Update1_Cockroach-24 727µs ± 2% 732µs ± 6% ~ (p=0.971 n=10+10) Update10_Cockroach-24 1.14ms ± 4% 1.15ms ± 3% ~ (p=0.211 n=9+10) Update100_Cockroach-24 4.53ms ±12% 4.46ms ± 6% ~ (p=0.796 n=10+10) Update1000_Cockroach-24 33.8ms ± 4% 32.9ms ± 3% -2.60% (p=0.019 n=10+10) Delete1_Cockroach-24 674µs ± 2% 671µs ± 2% ~ (p=0.796 n=10+10) Delete10_Cockroach-24 812µs ± 2% 828µs ± 2% +2.01% (p=0.003 n=9+10) Delete100_Cockroach-24 2.35ms ± 1% 2.39ms ± 5% ~ (p=0.094 n=9+9) Delete1000_Cockroach-24 17.0ms ± 4% 17.0ms ± 2% ~ (p=0.853 n=10+10) InterleavedSelect1000_Cockroach-24 52.7ms ± 4% 52.8ms ± 5% ~ (p=0.549 n=10+9) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% +0.03% (p=0.000 n=10+9) Insert1_Cockroach-24 36.6kB ± 0% 40.0kB ± 0% +9.25% (p=0.000 n=10+10) Insert10_Cockroach-24 85.2kB ± 0% 90.5kB ± 0% +6.14% (p=0.000 n=10+10) Insert100_Cockroach-24 538kB ± 0% 543kB ± 0% +0.96% (p=0.000 n=10+9) Insert1000_Cockroach-24 4.63MB ± 0% 4.64MB ± 0% +0.09% (p=0.002 n=10+10) Update1_Cockroach-24 62.2kB ± 0% 65.5kB ± 0% +5.37% (p=0.000 n=8+10) Update10_Cockroach-24 136kB ± 0% 142kB ± 0% +3.95% (p=0.000 n=10+9) Update100_Cockroach-24 863kB ± 0% 865kB ± 0% +0.29% (p=0.000 n=9+9) Update1000_Cockroach-24 7.15MB ± 0% 7.15MB ± 0% ~ (p=0.065 n=9+10) Delete1_Cockroach-24 54.0kB ± 0% 57.3kB ± 0% +6.15% (p=0.000 n=10+10) Delete10_Cockroach-24 64.4kB ± 0% 67.9kB ± 0% +5.49% (p=0.000 n=10+10) Delete100_Cockroach-24 173kB ± 0% 178kB ± 0% +2.87% (p=0.000 n=10+9) Delete1000_Cockroach-24 1.19MB ± 0% 1.14MB ± 0% -3.72% (p=0.000 n=9+10) InterleavedSelect1000_Cockroach-24 797kB ± 0% 797kB ± 0% ~ (p=0.811 n=10+10) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 386 ± 0% 395 ± 0% +2.33% (p=0.000 n=9+8) Insert10_Cockroach-24 612 ± 0% 622 ± 0% +1.63% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.64k ± 0% +0.36% (p=0.000 n=10+7) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% +0.05% (p=0.006 n=10+10) Update1_Cockroach-24 682 ± 0% 691 ± 0% +1.32% (p=0.000 n=10+10) Update10_Cockroach-24 1.01k ± 0% 1.02k ± 0% +0.96% (p=0.000 n=10+9) Update100_Cockroach-24 3.99k ± 0% 4.00k ± 0% +0.22% (p=0.000 n=9+9) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% +0.03% (p=0.000 n=9+9) Delete1_Cockroach-24 568 ± 0% 577 ± 0% +1.58% (p=0.000 n=10+8) Delete10_Cockroach-24 693 ± 0% 703 ± 0% +1.46% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.84k ± 0% +0.55% (p=0.000 n=9+10) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=0.954 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.586 n=10+7) ``` `master` vs proposer-evaluated KV. Takes a hit, as expected (prop-eval KV does more work as it first preps a batch, serializes and unserializes it, then commits (this is optimizable): ``` name old time/op new time/op delta Select1_Cockroach-24 78.1µs ± 2% 80.7µs ± 3% +3.26% (p=0.000 n=10+10) Insert1_Cockroach-24 507µs ± 5% 525µs ± 5% +3.41% (p=0.019 n=10+10) Insert10_Cockroach-24 662µs ± 2% 674µs ± 4% ~ (p=0.075 n=10+10) Insert100_Cockroach-24 1.74ms ± 5% 1.80ms ± 4% +3.69% (p=0.013 n=9+10) Insert1000_Cockroach-24 11.7ms ± 3% 11.6ms ± 3% ~ (p=0.436 n=10+10) Update1_Cockroach-24 727µs ± 4% 693µs ± 1% -4.67% (p=0.000 n=10+8) Update10_Cockroach-24 1.15ms ± 5% 1.14ms ± 6% ~ (p=0.579 n=10+10) Update100_Cockroach-24 4.42ms ± 6% 4.52ms ± 7% ~ (p=0.190 n=10+10) Update1000_Cockroach-24 32.9ms ± 3% 34.3ms ± 5% +4.04% (p=0.000 n=10+10) Delete1_Cockroach-24 675µs ± 3% 672µs ± 2% ~ (p=0.579 n=10+10) Delete10_Cockroach-24 799µs ± 2% 827µs ± 3% +3.46% (p=0.000 n=8+10) Delete100_Cockroach-24 2.38ms ± 3% 2.52ms ± 4% +6.16% (p=0.000 n=10+10) Delete1000_Cockroach-24 16.7ms ± 4% 17.7ms ± 3% +5.82% (p=0.000 n=10+10) InterleavedSelect1000_Cockroach-24 52.0ms ± 1% 51.9ms ± 2% ~ (p=0.274 n=10+8) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% +0.03% (p=0.000 n=8+9) Insert1_Cockroach-24 36.6kB ± 0% 39.5kB ± 0% +7.89% (p=0.000 n=9+10) Insert10_Cockroach-24 85.2kB ± 0% 91.2kB ± 0% +7.04% (p=0.000 n=10+10) Insert100_Cockroach-24 538kB ± 0% 559kB ± 0% +3.91% (p=0.000 n=10+10) Insert1000_Cockroach-24 4.63MB ± 0% 4.80MB ± 0% +3.54% (p=0.000 n=10+9) Update1_Cockroach-24 62.2kB ± 0% 65.4kB ± 0% +5.12% (p=0.000 n=10+10) Update10_Cockroach-24 136kB ± 0% 143kB ± 0% +5.09% (p=0.000 n=9+8) Update100_Cockroach-24 863kB ± 0% 883kB ± 0% +2.31% (p=0.000 n=8+10) Update1000_Cockroach-24 7.15MB ± 0% 7.33MB ± 0% +2.53% (p=0.000 n=10+10) Delete1_Cockroach-24 54.0kB ± 0% 56.8kB ± 0% +5.15% (p=0.000 n=10+10) Delete10_Cockroach-24 64.3kB ± 0% 68.7kB ± 0% +6.74% (p=0.000 n=10+10) Delete100_Cockroach-24 173kB ± 0% 194kB ± 0% +11.72% (p=0.000 n=10+10) Delete1000_Cockroach-24 1.19MB ± 0% 1.26MB ± 0% +5.98% (p=0.000 n=9+10) InterleavedSelect1000_Cockroach-24 797kB ± 0% 796kB ± 0% ~ (p=0.095 n=10+9) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 386 ± 0% 389 ± 0% +0.78% (p=0.000 n=8+9) Insert10_Cockroach-24 612 ± 0% 616 ± 0% +0.65% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.63k ± 0% +0.13% (p=0.000 n=9+6) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% -0.17% (p=0.000 n=10+10) Update1_Cockroach-24 682 ± 0% 685 ± 0% +0.44% (p=0.000 n=10+10) Update10_Cockroach-24 1.01k ± 0% 1.02k ± 0% +0.39% (p=0.000 n=9+8) Update100_Cockroach-24 3.99k ± 0% 3.99k ± 0% +0.12% (p=0.000 n=9+10) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% +0.04% (p=0.000 n=8+7) Delete1_Cockroach-24 568 ± 0% 571 ± 0% +0.65% (p=0.000 n=10+10) Delete10_Cockroach-24 692 ± 0% 697 ± 0% +0.64% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.83k ± 0% +0.31% (p=0.000 n=10+9) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=0.195 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.190 n=10+9) ``` Finally (just for completeness) this PR without proposer-eval'ed KV against the PR with proposer-eval'ed KV. ``` name old time/op new time/op delta Select1_Cockroach-24 80.7µs ± 3% 80.9µs ± 3% ~ (p=1.000 n=10+9) Insert1_Cockroach-24 525µs ± 5% 522µs ± 3% ~ (p=0.436 n=10+10) Insert10_Cockroach-24 674µs ± 4% 672µs ± 5% ~ (p=0.631 n=10+10) Insert100_Cockroach-24 1.80ms ± 4% 1.82ms ± 6% ~ (p=0.796 n=10+10) Insert1000_Cockroach-24 11.6ms ± 3% 11.9ms ± 1% +1.97% (p=0.008 n=10+9) Update1_Cockroach-24 693µs ± 1% 743µs ± 2% +7.29% (p=0.000 n=8+10) Update10_Cockroach-24 1.14ms ± 6% 1.18ms ± 5% ~ (p=0.075 n=10+10) Update100_Cockroach-24 4.52ms ± 7% 4.61ms ± 8% ~ (p=0.393 n=10+10) Update1000_Cockroach-24 34.3ms ± 5% 33.3ms ± 6% ~ (p=0.105 n=10+10) Delete1_Cockroach-24 672µs ± 2% 671µs ± 4% ~ (p=0.631 n=10+10) Delete10_Cockroach-24 827µs ± 3% 814µs ± 1% -1.59% (p=0.016 n=10+8) Delete100_Cockroach-24 2.52ms ± 4% 2.40ms ± 2% -4.94% (p=0.000 n=10+9) Delete1000_Cockroach-24 17.7ms ± 3% 17.2ms ± 3% -3.14% (p=0.009 n=10+10) InterleavedSelect1000_Cockroach-24 51.9ms ± 2% 51.8ms ± 3% ~ (p=0.633 n=8+10) name old alloc/op new alloc/op delta Select1_Cockroach-24 7.31kB ± 0% 7.31kB ± 0% ~ (p=0.246 n=9+10) Insert1_Cockroach-24 39.5kB ± 0% 40.0kB ± 0% +1.28% (p=0.000 n=10+10) Insert10_Cockroach-24 91.2kB ± 0% 90.4kB ± 0% -0.85% (p=0.000 n=10+10) Insert100_Cockroach-24 559kB ± 0% 543kB ± 0% -2.84% (p=0.000 n=10+9) Insert1000_Cockroach-24 4.80MB ± 0% 4.64MB ± 0% -3.36% (p=0.000 n=9+10) Update1_Cockroach-24 65.4kB ± 0% 65.5kB ± 0% +0.24% (p=0.000 n=10+10) Update10_Cockroach-24 143kB ± 0% 142kB ± 0% -0.93% (p=0.000 n=8+8) Update100_Cockroach-24 883kB ± 0% 865kB ± 0% -1.97% (p=0.000 n=10+8) Update1000_Cockroach-24 7.33MB ± 0% 7.15MB ± 0% -2.45% (p=0.000 n=10+10) Delete1_Cockroach-24 56.8kB ± 0% 57.4kB ± 0% +1.11% (p=0.000 n=10+10) Delete10_Cockroach-24 68.7kB ± 0% 67.9kB ± 0% -1.08% (p=0.000 n=10+10) Delete100_Cockroach-24 194kB ± 0% 178kB ± 0% -7.87% (p=0.000 n=10+10) Delete1000_Cockroach-24 1.26MB ± 0% 1.14MB ± 0% -9.15% (p=0.000 n=10+10) InterleavedSelect1000_Cockroach-24 796kB ± 0% 797kB ± 0% +0.05% (p=0.002 n=9+8) name old allocs/op new allocs/op delta Select1_Cockroach-24 111 ± 0% 111 ± 0% ~ (all equal) Insert1_Cockroach-24 389 ± 0% 395 ± 0% +1.54% (p=0.000 n=9+8) Insert10_Cockroach-24 616 ± 0% 622 ± 0% +0.97% (p=0.000 n=8+8) Insert100_Cockroach-24 2.63k ± 0% 2.64k ± 0% +0.24% (p=0.000 n=6+9) Insert1000_Cockroach-24 22.6k ± 0% 22.6k ± 0% +0.22% (p=0.000 n=10+10) Update1_Cockroach-24 685 ± 0% 691 ± 0% +0.88% (p=0.000 n=10+10) Update10_Cockroach-24 1.02k ± 0% 1.02k ± 0% +0.69% (p=0.000 n=8+7) Update100_Cockroach-24 3.99k ± 0% 4.00k ± 0% +0.13% (p=0.000 n=10+8) Update1000_Cockroach-24 31.6k ± 0% 31.6k ± 0% -0.02% (p=0.002 n=7+8) Delete1_Cockroach-24 571 ± 0% 578 ± 0% +1.10% (p=0.000 n=10+10) Delete10_Cockroach-24 697 ± 0% 704 ± 0% +0.98% (p=0.000 n=10+10) Delete100_Cockroach-24 1.83k ± 0% 1.84k ± 0% +0.28% (p=0.000 n=9+10) Delete1000_Cockroach-24 12.7k ± 0% 12.7k ± 0% ~ (p=1.000 n=10+10) InterleavedSelect1000_Cockroach-24 3.11k ± 0% 3.11k ± 0% ~ (p=0.079 n=9+10) ``` Block writer `master`, `PR, false` `PR, true`: ``` master BenchmarkBlockWriter 1435603 208981.4 ns/op PR wo BenchmarkBlockWriter 1508092 198933.8 ns/op with BenchmarkBlockWriter 1374589 218254.9 ns/op 1s: 4587.2/sec 4587.1/sec | 4349.7/sec 4349.4/sec | 4001.9/sec 4001.8/sec 2s: 4224.0/sec 4406.8/sec | 4330.0/sec 4339.8/sec | 3967.9/sec 3984.9/sec 3s: 4137.1/sec 4317.3/sec | 4185.0/sec 4288.4/sec | 3588.0/sec 3852.9/sec 4s: 4015.0/sec 4242.0/sec | 4204.7/sec 4267.5/sec | 3695.2/sec 3813.5/sec 5s: 4151.0/sec 4223.8/sec | 3983.2/sec 4210.7/sec | 3879.0/sec 3826.6/sec 6s: 4155.8/sec 4212.5/sec | 3851.0/sec 4150.9/sec | 3914.0/sec 3841.2/sec 7s: 4020.1/sec 4185.1/sec | 3895.2/sec 4114.4/sec | 3747.0/sec 3827.7/sec 8s: 3831.0/sec 4140.9/sec | 3807.9/sec 4076.1/sec | 3632.0/sec 3803.3/sec 9s: 3875.8/sec 4111.5/sec | 3914.8/sec 4058.2/sec | 3553.8/sec 3775.6/sec 10s: 3884.2/sec 4088.8/sec | 3987.2/sec 4051.1/sec | 3604.2/sec 3758.4/sec 11s: 4089.9/sec 4088.9/sec | 3911.0/sec 4038.4/sec | 3656.9/sec 3749.2/sec 12s: 3985.2/sec 4080.3/sec | 3901.9/sec 4027.0/sec | 3707.0/sec 3745.7/sec 13s: 4018.8/sec 4075.5/sec | 3804.0/sec 4009.9/sec | 3584.0/sec 3733.3/sec 14s: 3984.1/sec 4069.0/sec | 4010.9/sec 4010.0/sec | 3439.9/sec 3712.3/sec 15s: 3699.1/sec 4044.4/sec | 3665.1/sec 3987.0/sec | 3478.2/sec 3696.7/sec 16s: 3912.7/sec 4036.2/sec | 3764.0/sec 3973.1/sec | 3616.9/sec 3691.7/sec 17s: 3770.3/sec 4020.5/sec | 3747.0/sec 3959.8/sec | 3639.0/sec 3688.6/sec 18s: 3997.9/sec 4019.3/sec | 3903.9/sec 3956.7/sec | 3588.1/sec 3683.1/sec 19s: 3630.0/sec 3998.8/sec | 4115.8/sec 3965.0/sec | 3508.8/sec 3673.9/sec 20s: 4005.9/sec 3999.2/sec | 3870.2/sec 3960.3/sec | 3759.1/sec 3678.1/sec 21s: 3719.1/sec 3985.8/sec | 3618.1/sec 3944.0/sec | 3682.1/sec 3678.3/sec 22s: 3722.0/sec 3973.8/sec | 3970.1/sec 3945.2/sec | 3674.9/sec 3678.2/sec 23s: 3902.0/sec 3970.7/sec | 3441.9/sec 3923.3/sec | 3468.0/sec 3669.0/sec 24s: 3638.0/sec 3956.9/sec | 3684.9/sec 3913.4/sec | 3631.0/sec 3667.5/sec 25s: 6620.0/sec 4063.3/sec | 5308.0/sec 3969.2/sec | 3617.7/sec 3665.5/sec 26s: 6123.1/sec 4142.5/sec | 6115.1/sec 4051.7/sec | 3225.5/sec 3648.6/sec 27s: 6660.7/sec 4235.7/sec | 5974.1/sec 4122.8/sec | 5913.4/sec 3732.4/sec 28s: 6209.2/sec 4306.2/sec | 6431.0/sec 4205.2/sec | 5624.0/sec 3800.0/sec 29s: 6580.9/sec 4384.6/sec | 6189.8/sec 4273.7/sec | 6057.5/sec 3877.8/sec 30s: 6016.1/sec 4438.9/sec | 6489.3/sec 4347.5/sec | 5491.4/sec 3931.6/sec 31s: 6454.1/sec 4503.9/sec | 5094.9/sec 4371.6/sec | 6233.2/sec 4005.8/sec 32s: 5101.9/sec 4522.6/sec | 6427.0/sec 4435.8/sec | 5396.7/sec 4049.3/sec 33s: 6572.9/sec 4584.7/sec | 5962.2/sec 4482.0/sec | 5990.1/sec 4108.1/sec 34s: 4667.2/sec 4587.1/sec | 6078.0/sec 4529.0/sec | 5562.0/sec 4150.8/sec 35s: 6594.7/sec 4644.5/sec | 6301.0/sec 4579.6/sec | 5663.0/sec 4194.0/sec 36s: 6504.2/sec 4696.1/sec | 5936.9/sec 4617.3/sec | 5877.0/sec 4240.8/sec 37s: 6061.0/sec 4733.0/sec | 6466.9/sec 4667.3/sec | 5790.9/sec 4282.6/sec 38s: 6444.1/sec 4778.0/sec | 6123.9/sec 4705.6/sec | 5912.2/sec 4325.5/sec 39s: 6294.0/sec 4816.8/sec | 6248.1/sec 4745.1/sec | 5657.1/sec 4359.7/sec 40s: 6782.5/sec 4866.0/sec | 5872.2/sec 4773.3/sec | 5854.9/sec 4397.0/sec 41s: 9043.7/sec 4967.8/sec | 8606.9/sec 4866.8/sec | 5819.9/sec 4431.7/sec 42s: 8676.5/sec 5056.1/sec | 8308.8/sec 4948.7/sec | 5291.8/sec 4452.2/sec 43s: 8344.1/sec 5132.5/sec | 8517.3/sec 5031.7/sec | 6688.5/sec 4504.2/sec 44s: 8777.1/sec 5215.4/sec | 8169.6/sec 5103.0/sec | 8453.8/sec 4594.0/sec 45s: 8428.1/sec 5286.7/sec | 8810.7/sec 5185.4/sec | 7550.9/sec 4659.7/sec 46s: 8874.7/sec 5364.7/sec | 8286.6/sec 5252.8/sec | 8502.2/sec 4743.2/sec 47s: 8416.3/sec 5429.6/sec | 8596.9/sec 5323.9/sec | 6749.1/sec 4785.9/sec 48s: 8860.8/sec 5501.1/sec | 8173.3/sec 5383.2/sec | 8306.8/sec 4859.2/sec 49s: 8766.2/sec 5567.7/sec | 8589.9/sec 5448.7/sec | 8556.9/sec 4934.7/sec 50s: 8214.9/sec 5620.6/sec | 8577.9/sec 5511.3/sec | 6479.5/sec 4965.5/sec 51s: 8893.0/sec 5684.8/sec | 7346.0/sec 5547.2/sec | 8454.6/sec 5033.9/sec 52s: 8326.0/sec 5735.5/sec | 8664.9/sec 5607.2/sec | 6760.0/sec 5067.1/sec 53s: 8646.5/sec 5790.5/sec | 8677.1/sec 5665.1/sec | 8539.9/sec 5132.7/sec 54s: 8873.8/sec 5847.5/sec | 7655.2/sec 5701.9/sec | 6050.0/sec 5149.6/sec 55s: 8763.5/sec 5900.5/sec | 8749.0/sec 5757.3/sec | 8429.9/sec 5209.3/sec 56s: 6465.9/sec 5910.6/sec | 8723.9/sec 5810.3/sec | 8601.0/sec 5269.8/sec 57s: 3879.7/sec 5875.0/sec | 8659.0/sec 5860.3/sec | 3968.7/sec 5247.0/sec 58s: 2531.1/sec 5817.4/sec | 1449.9/sec 5784.2/sec | 6217.5/sec 5263.7/sec 59s: 4186.3/sec 5789.7/sec | 3661.2/sec 5748.2/sec | 8284.1/sec 5314.9/sec 1m0s: 7975.0/sec 5826.2/sec | 1211.1/sec 5672.6/sec | 2871.8/sec 5274.2/sec 1m1s: 6711.0/sec 5840.7/sec | 1431.9/sec 5603.1/sec | 4990.3/sec 5269.6/sec 1m2s: 4748.7/sec 5823.0/sec | 7576.7/sec 5635.0/sec | 6227.2/sec 5285.0/sec 1m3s: 5278.5/sec 5814.4/sec | 6698.6/sec 5651.8/sec | 8392.9/sec 5334.3/sec 1m4s: 2612.9/sec 5764.4/sec | 8542.8/sec 5697.0/sec | 4390.2/sec 5319.6/sec 1m5s: 4458.8/sec 5744.3/sec | 3849.8/sec 5668.6/sec | 15.0/sec 5238.0/sec 1m6s: 778.0/sec 5669.1/sec | 973.0/sec 5597.5/sec | 6057.5/sec 5250.4/sec 1m7s: 7397.8/sec 5694.9/sec | 6525.1/sec 5611.3/sec | 1583.9/sec 5195.7/sec 1m8s: 4206.3/sec 5673.0/sec | 5286.7/sec 5606.5/sec | 1670.1/sec 5143.8/sec 1m9s: 9329.0/sec 5726.0/sec | 145.0/sec 5527.4/sec | 7968.2/sec 5184.8/sec 1m10s: 4207.0/sec 5704.3/sec | 8178.6/sec 5565.2/sec | 6377.1/sec 5201.8/sec 1m11s: 467.1/sec 5630.5/sec | 5808.1/sec 5568.7/sec | 8555.2/sec 5249.0/sec 1m12s: 3249.5/sec 5597.5/sec | 3945.1/sec 5546.1/sec | 1160.3/sec 5192.3/sec 1m13s: 7547.9/sec 5624.2/sec |10219.6/sec 5610.1/sec | 753.0/sec 5131.5/sec 1m14s:10479.6/sec 5689.8/sec | 8955.5/sec 5655.3/sec | 4614.0/sec 5124.5/sec 1m15s: 2402.0/sec 5645.9/sec | 2536.8/sec 5613.8/sec | 818.0/sec 5067.1/sec 1m16s: 438.0/sec 5577.4/sec | 6229.0/sec 5621.8/sec | 1130.0/sec 5015.3/sec 1m17s: 3561.1/sec 5551.2/sec | 13.0/sec 5549.0/sec | 6255.3/sec 5031.4/sec 1m18s: 3588.8/sec 5526.1/sec | 9045.8/sec 5593.8/sec | 9972.2/sec 5094.7/sec 1m19s:10131.5/sec 5584.4/sec | 2488.7/sec 5554.5/sec | 7508.0/sec 5125.2/sec 1m20s: 1915.0/sec 5538.5/sec | 4666.1/sec 5543.4/sec | 9947.6/sec 5185.5/sec 1m21s: 1530.6/sec 5489.0/sec | 5259.1/sec 5539.9/sec | 8631.2/sec 5228.1/sec 1m22s: 5218.2/sec 5485.7/sec | 8594.8/sec 5577.2/sec | 2574.8/sec 5195.7/sec 1m23s: 1790.0/sec 5441.2/sec | 5812.0/sec 5580.0/sec | 6606.1/sec 5212.7/sec 1m24s: 1077.0/sec 5389.3/sec | 2105.3/sec 5538.6/sec | 2788.3/sec 5183.8/sec 1m25s: 9192.3/sec 5434.0/sec | 9297.8/sec 5582.9/sec | 7046.2/sec 5205.7/sec 1m26s: 2206.0/sec 5396.5/sec | 6761.8/sec 5596.6/sec | 2829.2/sec 5178.1/sec 1m27s: 8510.9/sec 5432.3/sec | 2130.1/sec 5556.7/sec | 7195.1/sec 5201.3/sec 1m28s: 7004.2/sec 5450.1/sec | 8941.0/sec 5595.2/sec | 4929.9/sec 5198.2/sec 1m29s:10563.6/sec 5507.6/sec | 6855.6/sec 5609.3/sec | 9913.1/sec 5251.2/sec 1m30s: 1207.0/sec 5459.8/sec | 2984.9/sec 5580.2/sec | 5823.7/sec 5257.5/sec 1m31s: 1888.0/sec 5420.5/sec | 9592.9/sec 5624.3/sec | 9037.5/sec 5299.1/sec 1m32s: 6090.1/sec 5427.8/sec | 8368.9/sec 5654.1/sec | 1075.9/sec 5253.2/sec 1m33s: 1273.0/sec 5383.1/sec | 9538.1/sec 5695.9/sec | 2864.3/sec 5227.5/sec 1m34s: 1743.9/sec 5344.4/sec | 5547.4/sec 5694.3/sec | 5787.0/sec 5233.4/sec 1m35s: 6407.9/sec 5355.6/sec | 1070.1/sec 5645.6/sec | 5361.6/sec 5234.8/sec 1m36s: 9923.0/sec 5403.2/sec | 9663.3/sec 5687.5/sec | 4910.0/sec 5231.4/sec 1m37s: 4074.8/sec 5389.5/sec | 6670.1/sec 5697.6/sec | 5113.5/sec 5230.2/sec 1m38s: 9495.4/sec 5431.4/sec | 5261.2/sec 5693.1/sec | 8203.4/sec 5260.5/sec 1m39s: 4153.8/sec 5418.5/sec | 22.0/sec 5635.9/sec | 4737.9/sec 5255.2/sec 1m40s: 103.0/sec 5365.3/sec | 6668.1/sec 5646.2/sec | 7005.8/sec 5272.8/sec 1m41s: 286.0/sec 5315.1/sec | 1530.9/sec 5605.4/sec | 5857.7/sec 5278.5/sec 1m42s: 7923.5/sec 5340.6/sec | 9456.7/sec 5643.2/sec | 8831.6/sec 5313.4/sec 1m43s:10307.9/sec 5388.8/sec | 8832.2/sec 5674.1/sec | 884.0/sec 5270.4/sec 1m44s: 4427.9/sec 5379.6/sec | 6880.4/sec 5685.7/sec | 935.0/sec 5228.7/sec 1m45s: 1116.0/sec 5339.0/sec | 264.0/sec 5634.1/sec | 4876.8/sec 5225.3/sec 1m46s: 9290.0/sec 5376.3/sec | 85.0/sec 5581.8/sec | 6945.1/sec 5241.6/sec 1m47s:10097.9/sec 5420.4/sec | 1739.3/sec 5545.9/sec | 4116.4/sec 5231.0/sec 1m48s: 2953.8/sec 5397.6/sec | 7145.1/sec 5560.7/sec | 4073.2/sec 5220.3/sec 1m49s: 2462.0/sec 5370.6/sec | 3741.0/sec 5544.0/sec | 6090.8/sec 5228.3/sec 1m50s: 6493.2/sec 5380.8/sec | 6756.6/sec 5555.0/sec | 1162.9/sec 5191.4/sec 1m51s: 5867.9/sec 5385.2/sec | 9214.1/sec 5588.0/sec | 9444.3/sec 5229.7/sec 1m52s: 80.0/sec 5337.9/sec | 7630.3/sec 5606.2/sec | 8124.0/sec 5255.5/sec 1m53s: 66.0/sec 5291.2/sec | 22.0/sec 5556.8/sec | 1259.9/sec 5220.1/sec 1m54s:5144.2/sec 5289.9/sec | 8677.7/sec 5584.1/sec | 7879.3/sec 5243.5/sec 1m55s: 10304.2/sec 5333.5/sec | 5515.0/sec 5583.5/sec | 15.0/sec 5198.0/sec 1m56s: 5323.0/sec 5333.4/sec | 4051.9/sec 5570.3/sec | 8838.8/sec 5229.4/sec 1m57s: 48.0/sec 5288.3/sec | 6286.7/sec 5576.5/sec | 35.0/sec 5185.0/sec 1m58s: 5770.6/sec 5292.3/sec | 9040.7/sec 5605.8/sec | 126.0/sec 5142.1/sec 1m59s: 3523.4/sec 5277.5/sec | 9035.0/sec 5634.6/sec | 2184.4/sec 5117.3/sec 2m0s: 8119.7/sec 5301.2/sec | 7646.5/sec 5651.4/sec | 7637.9/sec 5138.3/sec 2m1s: 4755.7/sec 5296.7/sec | 3334.8/sec 5632.3/sec | 1518.0/sec 5108.4/sec 2m2s:10283.7/sec 5337.5/sec | 755.0/sec 5592.3/sec | 4609.2/sec 5104.3/sec 2m3s: 5964.9/sec 5342.6/sec | 6245.0/sec 5597.6/sec | 5944.8/sec 5111.1/sec 2m4s: 2230.7/sec 5317.5/sec | 7813.5/sec 5615.5/sec | 5123.2/sec 5111.2/sec 2m5s: 5116.7/sec 5315.9/sec | 4813.5/sec 5609.0/sec | 6269.3/sec 5120.5/sec 2m6s: 3105.0/sec 5298.4/sec | 5171.3/sec 5605.6/sec | 4086.5/sec 5112.3/sec 2m7s: 6890.8/sec 5310.9/sec | 4732.9/sec 5598.7/sec | 3453.0/sec 5099.2/sec 2m8s: 1377.0/sec 5280.2/sec | 2662.3/sec 5575.8/sec | 42.0/sec 5059.7/sec 2m9s: 48.0/sec 5239.6/sec | 6369.8/sec 5581.9/sec | 5075.4/sec 5059.8/sec 2m10s: 5518.2/sec 5241.8/sec | 5810.2/sec 5583.7/sec | 5711.2/sec 5064.8/sec 2m11s: 9196.1/sec 5271.9/sec | 4373.2/sec 5574.4/sec | 1813.0/sec 5040.0/sec 2m12s: 1666.9/sec 5244.6/sec | 7006.9/sec 5585.3/sec | 3148.4/sec 5025.7/sec 2m13s: 6160.3/sec 5251.5/sec | 0.0/sec 5543.3/sec | 8122.3/sec 5048.9/sec 2m14s: 7106.3/sec 5265.4/sec | 0.0/sec 5501.9/sec | 4414.4/sec 5044.2/sec 2m15s: 4772.5/sec 5261.7/sec | 2.0/sec 5461.2/sec | 8962.5/sec 5073.2/sec 2m16s: 4069.8/sec 5253.0/sec | 1009.0/sec 5428.5/sec | 33.0/sec 5036.2/sec 2m17s: 5765.1/sec 5256.7/sec | 3482.2/sec 5414.3/sec | 41.0/sec 4999.7/sec 2m18s: 8875.2/sec 5282.9/sec | 9462.9/sec 5443.6/sec | 2424.8/sec 4981.1/sec 2m19s: 5182.5/sec 5282.2/sec | 9167.0/sec 5470.4/sec | 7051.9/sec 4996.0/sec 2m20s: 1634.1/sec 5256.1/sec |10603.1/sec 5507.0/sec | 5391.7/sec 4998.8/sec 2m21s: 323.0/sec 5221.2/sec | 1704.8/sec 5480.1/sec | 1377.0/sec 4973.1/sec 2m22s: 5411.2/sec 5222.5/sec | 8832.5/sec 5503.7/sec | 2668.0/sec 4956.9/sec 2m23s: 1981.0/sec 5199.8/sec | 712.0/sec 5470.2/sec | 6169.5/sec 4965.3/sec 2m24s: 3647.6/sec 5189.0/sec |11296.1/sec 5510.6/sec | 3489.7/sec 4955.1/sec 2m25s: 166.0/sec 5154.4/sec | 2499.8/sec 5489.9/sec | 5114.0/sec 4956.2/sec 2m26s: 5974.4/sec 5160.0/sec | 6934.7/sec 5499.8/sec | 6929.5/sec 4969.7/sec 2m27s: 1527.1/sec 5135.3/sec | 2523.4/sec 5479.5/sec | 731.9/sec 4940.9/sec 2m28s: 359.0/sec 5103.0/sec | 54.0/sec 5442.8/sec | 0.0/sec 4907.5/sec 2m29s: 8377.1/sec 5125.0/sec | 6178.9/sec 5447.8/sec | 680.0/sec 4879.1/sec 2m30s: 4546.7/sec 5121.2/sec | 6312.5/sec 5453.5/sec | 8254.5/sec 4901.6/sec 2m31s: 3130.9/sec 5108.0/sec | 9287.2/sec 5478.9/sec | 1600.0/sec 4879.8/sec 2m32s: 1238.9/sec 5082.5/sec | 10.0/sec 5443.0/sec | 2549.9/sec 4864.4/sec 2m33s: 8487.0/sec 5104.8/sec | 2116.2/sec 5421.2/sec | 9079.3/sec 4892.0/sec 2m34s: 42.0/sec 5071.9/sec | 5653.3/sec 5422.7/sec | 1373.9/sec 4869.1/sec 2m35s: 73.0/sec 5039.6/sec |10985.9/sec 5458.6/sec | 1715.9/sec 4848.8/sec 2m36s: 8468.5/sec 5061.6/sec | 5165.2/sec 5456.7/sec | 3184.1/sec 4838.1/sec 2m37s: 7946.5/sec 5080.0/sec | 1527.3/sec 5431.7/sec | 8996.7/sec 4864.6/sec 2m38s: 2291.8/sec 5062.3/sec | 5349.0/sec 5431.2/sec | 5937.5/sec 4871.4/sec 2m39s:10515.7/sec 5096.6/sec | 4281.7/sec 5424.0/sec | 7707.2/sec 4889.2/sec 2m40s: 7624.0/sec 5112.4/sec | 3941.0/sec 5414.7/sec | 41.0/sec 4858.9/sec 2m41s: 6627.3/sec 5121.8/sec | 7552.6/sec 5428.0/sec | 5506.6/sec 4862.9/sec 2m42s: 779.0/sec 5095.0/sec | 6501.1/sec 5434.6/sec | 2935.9/sec 4851.1/sec 2m43s: 6369.6/sec 5102.9/sec | 7616.9/sec 5448.0/sec |10298.3/sec 4884.5/sec 2m44s: 1653.0/sec 5081.8/sec | 2503.4/sec 5430.0/sec | 5839.0/sec 4890.3/sec 2m45s: 2058.1/sec 5063.5/sec | 33.0/sec 5397.3/sec |10947.2/sec 4927.0/sec 2m46s: 9075.9/sec 5087.7/sec | 9044.7/sec 5419.3/sec | 4132.5/sec 4922.2/sec 2m47s:10425.3/sec 5119.6/sec | 3219.8/sec 5406.1/sec | 8988.0/sec 4946.6/sec 2m48s: 3264.3/sec 5108.6/sec | 1196.1/sec 5381.1/sec |10646.2/sec 4980.5/sec 2m49s: 2728.0/sec 5094.5/sec | 4809.8/sec 5377.7/sec | 2066.0/sec 4963.2/sec 2m50s:10931.3/sec 5128.8/sec | 6443.3/sec 5383.9/sec | 1873.0/sec 4945.1/sec 2m51s: 6323.5/sec 5135.8/sec | 164.0/sec 5353.4/sec | 4180.2/sec 4940.6/sec 2m52s: 1607.0/sec 5115.3/sec | 6430.5/sec 5359.7/sec | 2776.9/sec 4928.0/sec 2m53s: 982.0/sec 5091.4/sec | 9842.2/sec 5385.6/sec |11246.5/sec 4964.5/sec 2m54s: 7885.9/sec 5107.5/sec | 1940.9/sec 5365.8/sec | 3676.7/sec 4957.1/sec 2m55s: 1905.9/sec 5089.2/sec |10557.4/sec 5395.5/sec | 9840.8/sec 4985.0/sec 2m56s: 1937.1/sec 5071.3/sec | 7563.9/sec 5407.8/sec | 5596.0/sec 4988.5/sec 2m57s: 1679.0/sec 5052.1/sec | 1272.1/sec 5384.4/sec | 4720.2/sec 4987.0/sec 2m58s: 4255.4/sec 5047.6/sec | 9745.7/sec 5408.9/sec | 1282.2/sec 4966.2/sec 2m59s: 8426.2/sec 5066.5/sec | 7192.5/sec 5418.9/sec | 0.0/sec 4938.4/sec 3m0s: 1167.0/sec 5044.8/sec | 1870.0/sec 5399.2/sec | 2833.1/sec 4926.7/sec 3m1s: 3522.1/sec 5036.4/sec | 48.0/sec 5369.6/sec | 56.0/sec 4899.8/sec 3m2s: 3810.0/sec 5029.7/sec | 3947.4/sec 5361.8/sec | 6804.3/sec 4910.3/sec 3m3s: 3109.1/sec 5019.2/sec | 4846.9/sec 5359.0/sec | 5008.5/sec 4910.8/sec 3m4s: 57.0/sec 4992.2/sec | 44.0/sec 5330.1/sec | 4704.1/sec 4909.7/sec 3m5s: 4918.1/sec 4991.8/sec | 8990.2/sec 5349.9/sec | 5397.7/sec 4912.3/sec 3m6s: 9671.3/sec 5017.0/sec | 3887.1/sec 5342.0/sec | 55.0/sec 4886.2/sec 3m7s: 130.0/sec 4990.9/sec | 8050.1/sec 5356.5/sec | 5212.2/sec 4888.0/sec 3m8s: 199.0/sec 4965.4/sec | 8238.1/sec 5371.8/sec | 3365.8/sec 4879.9/sec 3m9s: 836.0/sec 4943.5/sec | 9291.2/sec 5392.6/sec | 7874.5/sec 4895.7/sec 3m10s: 1827.0/sec 4927.1/sec | 19.0/sec 5364.3/sec | 2081.9/sec 4880.9/sec 3m11s: 4937.3/sec 4927.2/sec | 4660.5/sec 5360.6/sec | 41.0/sec 4855.6/sec 3m12s: 336.0/sec 4903.3/sec | 5277.5/sec 5360.2/sec | 6003.1/sec 4861.5/sec 3m13s: 2059.3/sec 4888.5/sec | 1565.1/sec 5340.5/sec | 1630.1/sec 4844.8/sec 3m14s:11245.9/sec 4921.3/sec | 6013.7/sec 5344.0/sec | 1080.9/sec 4825.4/sec 3m15s: 2425.9/sec 4908.5/sec | 2092.2/sec 5327.3/sec | 1754.1/sec 4809.6/sec 3m16s:10817.1/sec 4938.6/sec | 5876.5/sec 5330.1/sec | 7598.4/sec 4823.9/sec 3m17s: 1429.7/sec 4920.8/sec | 6989.3/sec 5338.5/sec | 8318.6/sec 4841.6/sec 3m18s: 4555.1/sec 4919.0/sec |10488.4/sec 5364.5/sec | 4193.9/sec 4838.3/sec 3m19s: 2661.3/sec 4907.6/sec | 3623.0/sec 5355.8/sec | 3804.2/sec 4833.1/sec 3m20s: 17.0/sec 4883.2/sec | 5813.5/sec 5358.1/sec | 4302.8/sec 4830.5/sec 3m21s: 2624.1/sec 4872.0/sec | 8798.3/sec 5375.2/sec | 5396.1/sec 4833.3/sec 3m22s: 2060.1/sec 4858.0/sec | 4410.4/sec 5370.4/sec | 3108.5/sec 4824.8/sec 3m23s: 8427.1/sec 4875.6/sec | 9463.4/sec 5390.6/sec | 4465.0/sec 4823.0/sec 3m24s: 5281.8/sec 4877.6/sec | 3938.9/sec 5383.4/sec | 122.0/sec 4799.9/sec 3m25s: 1801.8/sec 4862.6/sec | 10357.4/sec 5407.7/sec| 8710.2/sec 4819.0/sec 3m26s: 5574.2/sec 4866.1/sec | 9170.2/sec 5426.0/sec | 2832.1/sec 4809.4/sec 3m27s: 6970.0/sec 4876.2/sec | 62.0/sec 5400.1/sec | 5623.2/sec 4813.3/sec 3m28s: 4561.1/sec 4874.7/sec | 4262.5/sec 5394.6/sec | 3071.6/sec 4804.9/sec 3m29s: 5857.8/sec 4879.4/sec | 828.0/sec 5372.7/sec | 0.0/sec 4781.9/sec 3m30s: 1756.0/sec 4864.5/sec | 3004.9/sec 5361.5/sec | 0.0/sec 4759.2/sec 3m31s: 3234.0/sec 4856.8/sec | 3952.0/sec 5354.8/sec | 0.0/sec 4736.6/sec 3m32s: 3143.0/sec 4848.7/sec | 3447.0/sec 5345.8/sec | 0.0/sec 4714.3/sec 3m33s: 2955.2/sec 4839.8/sec | 2388.0/sec 5331.9/sec | 0.0/sec 4692.1/sec 3m34s: 2099.6/sec 4827.0/sec | 5861.5/sec 5334.4/sec | 4770.4/sec 4692.5/sec 3m35s: 4698.0/sec 4826.4/sec | 7248.3/sec 5343.3/sec | 8676.7/sec 4711.0/sec 3m36s: 7993.7/sec 4841.1/sec | 3046.1/sec 5332.6/sec | 88.0/sec 4689.6/sec 3m37s: 7314.4/sec 4852.5/sec | 8254.4/sec 5346.1/sec | 5004.5/sec 4691.1/sec 3m38s: 9046.5/sec 4871.7/sec | 4847.6/sec 5343.8/sec | 2257.1/sec 4679.9/sec 3m39s: 9369.4/sec 4892.3/sec | 8348.0/sec 5357.5/sec | 2752.0/sec 4671.1/sec 3m40s: 4950.4/sec 4892.5/sec | 62.0/sec 5333.5/sec | 6111.4/sec 4677.7/sec 3m41s: 61.0/sec 4870.7/sec | 2671.0/sec 5321.4/sec | 407.1/sec 4658.3/sec 3m42s: 7474.4/sec 4882.4/sec | 2607.3/sec 5309.2/sec | 57.0/sec 4637.6/sec 3m43s: 7187.7/sec 4892.7/sec | 5913.5/sec 5311.9/sec | 4693.3/sec 4637.9/sec 3m44s: 7027.2/sec 4902.2/sec | 4433.3/sec 5308.0/sec | 2793.8/sec 4629.6/sec 3m45s: 3752.3/sec 4897.1/sec |10589.6/sec 5331.5/sec | 8715.7/sec 4647.8/sec 3m46s: 9824.8/sec 4918.9/sec | 4915.1/sec 5329.6/sec | 4384.0/sec 4646.6/sec 3m47s:10184.8/sec 4942.1/sec | 8602.4/sec 5344.0/sec | 106.0/sec 4626.6/sec 3m48s: 2577.9/sec 4931.8/sec | 1374.0/sec 5326.6/sec | 9405.0/sec 4647.6/sec 3m49s: 8359.3/sec 4946.7/sec | 2558.1/sec 5314.5/sec | 3944.3/sec 4644.5/sec 3m50s: 8608.3/sec 4962.7/sec | 966.0/sec 5295.6/sec | 5170.7/sec 4646.8/sec 3m51s: 3402.7/sec 4955.9/sec | 5807.4/sec 5297.8/sec | 3573.7/sec 4642.1/sec 3m52s: 5969.5/sec 4960.3/sec | 6293.8/sec 5302.1/sec | 9580.1/sec 4663.4/sec 3m53s: 4655.6/sec 4959.0/sec | 330.1/sec 5280.8/sec |10790.9/sec 4689.7/sec 3m54s: 4637.9/sec 4957.6/sec | 1899.0/sec 5266.4/sec | 5911.4/sec 4694.9/sec 3m55s: 7502.2/sec 4968.4/sec | 5942.4/sec 5269.2/sec | 3858.4/sec 4691.4/sec 3m56s: 3467.5/sec 4962.1/sec | 8172.9/sec 5281.5/sec | 8306.7/sec 4706.7/sec 3m57s: 1986.1/sec 4949.5/sec | 500.0/sec 5261.4/sec | 2844.8/sec 4698.8/sec 3m58s: 6150.3/sec 4954.5/sec | 6793.5/sec 5267.8/sec | 0.0/sec 4679.1/sec 3m59s: 1767.9/sec 4941.2/sec | 9447.6/sec 5285.3/sec | 0.0/sec 4659.5/sec 4m0s: 0.0/sec 4920.6/sec | 4337.1/sec 5281.3/sec | 5272.7/sec 4662.1/sec 4m1s: 0.0/sec 4900.2/sec | 1792.2/sec 5266.9/sec | 2052.0/sec 4651.3/sec 4m2s: 0.0/sec 4880.0/sec | 4982.0/sec 5265.7/sec | 40.0/sec 4632.2/sec 4m3s: 0.0/sec 4859.9/sec | 8689.0/sec 5279.8/sec | 7462.2/sec 4643.8/sec 4m4s: 5126.8/sec 4861.0/sec | 3963.1/sec 5274.4/sec | 6461.0/sec 4651.3/sec 4m5s: 9416.8/sec 4879.6/sec | 4915.1/sec 5272.9/sec | 3797.6/sec 4647.8/sec 4m6s: 5596.7/sec 4882.5/sec | 5439.8/sec 5273.6/sec | 6717.2/sec 4656.2/sec 4m7s: 7137.0/sec 4891.6/sec | 2062.2/sec 5260.6/sec | 5986.5/sec 4661.6/sec 4m8s: 2900.1/sec 4883.6/sec | 4199.6/sec 5256.3/sec | 5423.3/sec 4664.7/sec 4m9s: 2352.7/sec 4873.4/sec | 5207.5/sec 5256.1/sec | 5621.3/sec 4668.5/sec 4m10s: 7146.7/sec 4882.5/sec | 49.0/sec 5235.3/sec | 59.0/sec 4650.1/sec 4m11s: 7591.3/sec 4893.3/sec | 3108.5/sec 5226.8/sec | 4744.5/sec 4650.5/sec 4m12s: 3887.8/sec 4889.3/sec | 2972.8/sec 5217.9/sec | 5453.8/sec 4653.6/sec 4m13s: 0.0/sec 4870.0/sec | 5755.6/sec 5220.0/sec | 5422.1/sec 4656.7/sec 4m14s: 1462.2/sec 4856.6/sec | 1625.7/sec 5205.9/sec | 3522.7/sec 4652.2/sec 4m15s: 4982.9/sec 4857.1/sec | 368.0/sec 5186.9/sec |10206.9/sec 4674.1/sec 4m16s:10435.8/sec 4878.9/sec |11150.1/sec 5210.2/sec | 617.9/sec 4658.3/sec 4m17s: 5842.5/sec 4882.6/sec | 2585.7/sec 5200.0/sec | 3725.3/sec 4654.7/sec 4m18s: 3380.2/sec 4876.8/sec |10330.0/sec 5219.9/sec | 5500.1/sec 4658.0/sec 4m19s: 6659.0/sec 4883.7/sec | 7819.7/sec 5229.9/sec | 6104.0/sec 4663.6/sec 4m20s: 8776.2/sec 4898.6/sec | 0.0/sec 5209.8/sec | 4216.6/sec 4661.8/sec 4m21s: 6155.8/sec 4903.5/sec | 6136.3/sec 5213.3/sec | 7253.9/sec 4671.8/sec 4m22s: 3866.0/sec 4899.5/sec | 3681.5/sec 5207.5/sec | 29.0/sec 4654.1/sec 4m23s: 8734.5/sec 4914.1/sec | 8508.6/sec 5220.0/sec | 2136.9/sec 4644.5/sec 4m24s: 3044.3/sec 4907.0/sec |10632.0/sec 5240.5/sec |10642.7/sec 4667.2/sec 4m25s: 5469.2/sec 4909.1/sec | 3237.3/sec 5233.0/sec | 3578.8/sec 4663.1/sec 4m26s: 141.0/sec 4891.2/sec | 2676.2/sec 5223.3/sec | 84.0/sec 4645.9/sec 4m27s: 9001.3/sec 4906.6/sec | 5368.3/sec 5223.9/sec | 9260.7/sec 4663.2/sec 4m28s: 2234.9/sec 4896.6/sec | 2853.3/sec 5215.0/sec | 2918.7/sec 4656.6/sec 4m29s: 9095.2/sec 4912.2/sec | 5955.1/sec 5217.8/sec | 2949.2/sec 4650.3/sec 4m30s: 1032.0/sec 4897.9/sec | 2187.2/sec 5206.6/sec |10807.0/sec 4673.1/sec 4m31s: 4271.4/sec 4895.5/sec | 4461.0/sec 5203.8/sec | 5457.1/sec 4676.0/sec 4m32s: 1673.0/sec 4883.7/sec | 4818.8/sec 5202.4/sec | 6558.1/sec 4682.9/sec 4m33s: 3209.4/sec 4877.6/sec | 5941.2/sec 5205.1/sec | 5019.6/sec 4684.1/sec 4m34s: 288.0/sec 4860.8/sec | 2781.2/sec 5196.3/sec | 2193.2/sec 4675.1/sec 4m35s: 3475.2/sec 4855.8/sec | 1115.3/sec 5181.4/sec | 7251.0/sec 4684.4/sec 4m36s: 8060.5/sec 4867.4/sec | 1429.7/sec 5167.8/sec | 296.0/sec 4668.5/sec 4m37s: 85.0/sec 4850.1/sec | 4989.5/sec 5167.2/sec | 5162.4/sec 4670.3/sec 4m38s: 6838.7/sec 4857.3/sec | 1986.1/sec 5155.7/sec | 3189.4/sec 4665.0/sec 4m39s: 3861.7/sec 4853.7/sec | 98.0/sec 5137.6/sec | 8847.6/sec 4680.0/sec 4m40s: 8261.0/sec 4865.9/sec | 3867.6/sec 5133.1/sec | 2022.0/sec 4670.5/sec 4m41s: 3799.3/sec 4862.1/sec | 4147.5/sec 5129.6/sec | 1072.0/sec 4657.7/sec 4m42s: 1449.2/sec 4850.0/sec | 8232.9/sec 5140.6/sec | 8564.0/sec 4671.5/sec 4m43s: 3237.0/sec 4844.3/sec | 2193.8/sec 5130.2/sec | 2522.2/sec 4663.9/sec 4m44s: 2437.2/sec 4835.8/sec | 0.0/sec 5112.1/sec | 2106.9/sec 4654.9/sec 4m45s: 5582.3/sec 4838.4/sec | 0.0/sec 5094.2/sec | 4277.9/sec 4653.6/sec 4m46s: 4077.4/sec 4835.8/sec | 1699.9/sec 5082.3/sec | 6279.3/sec 4659.3/sec 4m47s: 1518.0/sec 4824.2/sec | 9092.4/sec 5096.2/sec | 5723.4/sec 4663.0/sec 4m48s: 5973.9/sec 4828.2/sec | 1382.1/sec 5083.4/sec | 2328.0/sec 4654.9/sec 4m49s: 1477.9/sec 4816.6/sec | 3623.8/sec 5078.3/sec |10018.7/sec 4673.5/sec 4m50s: 19.0/sec 4800.1/sec | 7984.6/sec 5088.3/sec | 6391.7/sec 4679.4/sec 4m51s: 1320.0/sec 4788.1/sec | 1828.9/sec 5077.1/sec | 7006.3/sec 4687.4/sec 4m52s: 9922.4/sec 4805.7/sec | 1778.0/sec 5065.8/sec | 507.0/sec 4673.1/sec 4m53s: 5176.3/sec 4806.9/sec | 39.0/sec 5048.7/sec | 646.0/sec 4659.3/sec 4m54s: 197.0/sec 4791.3/sec | 1036.1/sec 5035.0/sec | 1559.8/sec 4648.8/sec 4m55s: 22.0/sec 4775.1/sec | 8240.7/sec 5045.9/sec | 196.0/sec 4633.7/sec 4m56s:10154.1/sec 4793.3/sec | 9409.3/sec 5060.6/sec | 5429.9/sec 4636.4/sec 4m57s: 2641.1/sec 4786.0/sec | 8583.7/sec 5072.5/sec | 880.0/sec 4623.7/sec 4m58s:10420.8/sec 4804.9/sec | 692.0/sec 5057.8/sec | 935.0/sec 4611.3/sec 4m59s: 599.9/sec 4790.9/sec | 789.0/sec 5043.5/sec | 0.0/sec 4595.9/sec 5m0s: 3067.3/sec 4785.1/sec | 34.0/sec 5026.8/sec | 329.1/sec 4581.7/sec
This method was previously called both before and after raft, and the after-raft logic relied on fields that were set in the ReplicatedProposalData by the before-raft logic. This caused inconsistencies when cockroachdb#10327 was deployed without freeze-cluster. Now, evaluateProposal is called only once, either before or after raft, and the post-raft logic handles the command in whichever fashion is appropriate for the version that proposed it. Fixes cockroachdb#10602
This method was previously called both before and after raft, and the after-raft logic relied on fields that were set in the ReplicatedProposalData by the before-raft logic. This caused inconsistencies when cockroachdb#10327 was deployed without freeze-cluster. Now, evaluateProposal is called only once, either before or after raft, and the post-raft logic handles the command in whichever fashion is appropriate for the version that proposed it. Fixes cockroachdb#10602
This method was previously called both before and after raft, and the after-raft logic relied on fields that were set in the ReplicatedProposalData by the before-raft logic. This caused inconsistencies when cockroachdb#10327 was deployed without freeze-cluster. Now, evaluateProposal is called only once, either before or after raft, and the post-raft logic handles the command in whichever fashion is appropriate for the version that proposed it. Fixes cockroachdb#10602
storage: fork-lift proposer-evaluated KV
Add experimental proposer-evaluated KV gated behind the environment variable
COCKROACH_PROPOSER_EVALUATED_KV. When set to a truthy value, Raft proposalsare evaluated and the resulting RocksDB
WriteBatchsubmitted to Raft alongwith some auxiliary metadata. The result of the evaluation is only stored in
the pending command on the proposer, and returned to the waiting client after
the
WriteBatchhas been applied.Introduce a natural failfast path for (most) proposals returning an error.
Instead of proposing, waiting for Raft, and only then receiving an error,
proposals which do not lead to a state change receive their error directly
when the proposal is evaluated, upstream of Raft. Only errors which still
want to persist data (for instance,
*TransactionRetryErrorwhen intentswere laid down) go through the whole proposal, with the client receiving
the error after the associated
Batchcommits.While proposer-evaluated KV is now ready for review, preliminary testing and
benchmarking, the current implementation is incomplete and incorrect:
Leaseacquisition is not special-cased, meaning that lease state may beclobbered freely when non-leaseholders propose a lease request based on stale
data. This needs to be fixed but it also shows that we don't stress that
scenario sufficiently in testing yet.
they were proposed (which is necessary).
CommandQueuedoes not account for internal keys accessed by overlappingcommands correctly (this is tracked in [DNM] add actual keys read to command queue #10084), which in principle also lead
to anomalies which should be exposed by testing and addressed.
Instead, every command inserts a span that covers everything. Horrible
for performance, great for correctness; [DNM] add actual keys read to command queue #10084 needs to address this.
TestingCommandFilterneeds to be refactored to be an explicit interceptorfor the pre-Raft stage of commands. Tests were fixed up enough to pass with
proposer-evaluated KV as well, but it's possible that some tests don't test
what they used to.
Benchmark results for non-proposer-evaluated-KV against
masterbelow.Note that all of them have the "correctness hack" in
(*Replica).beginCmdsdisabled; not doing that should give much worse results but we don't care about
those in practice
We pay a small fee in allocations, but nothing appears outrageous. So from
a performance perspective, this PR seems safe to merge:
mastervs proposer-evaluated KV. Takes a hit, as expected (prop-eval KV doesmore work as it first preps a batch, serializes and unserializes it, then
commits (this is optimizable):
Finally (just for completeness) this PR without proposer-eval'ed KV against the
PR with proposer-eval'ed KV.
Block writer
master,PR, falsePR, true: