20.2 → 21.1 roachperf benchmark regressions

[erikgrinaker]

Tracking issue for 20.2 → 21.1 roachperf benchmark regressions.

• bench: tpccbench regression between Feb 12th and 13th #62078: tpccbench regression between Feb 12th and 13th
  • Cause: separated intents *: configure the use of separated intents in a cluster #59829
  • Partial fix: db: optimize levelIter for non-matching bloom filter pebble#1091 vendor: bump Pebble to 92fbeeab #62264
  • Backport: [release-21.1] vendor: Bump pebble to b49f2eeba24b #62676
• kv: investigate kv95 regression around Feb 18 #62148: investigate kv95 regression around Feb 18
  • Cause: always-on tracing tracing: propagate non-recording spans across rpc boundaries #59992
  • Fix: sql: only create real spans when session tracing/sampling #61777 [wip,dnm] tracing: peephole optimizations #62227
  • Backport: release-21.1: sql: only create real spans when session tracing/sampling #61965
• kv: investigate kv95 performance after Feb 18 #62359: investigate kv95 performance after Feb 18
  • kvfollowerreadsccl: excessive overhead in canSendToFollower #62447: kvfollowerreadsccl: excessive overhead in canSendToFollower
    • Cause: follower reads kv: route present-time reads to global_read follower replicas #59571
    • Fix: kvccl: re-order enterprise check in canSendToFollower #62465 utilccl,kvccl: improve performance when checking enterprise features #62498
    • Backport: release-21.1: kvccl: re-order enterprise check in canSendToFollower #62606
  • catalog protoutil.Clone()
    • Cause: catalog, catalogkv: add and use descriptor builder #61429
    • Fix: sql: add tabledesc.NewUnsafeImmutable constructor #62388
    • Backport: release-21.1: sql: add tabledesc.NewUnsafeImmutable constructor #62616
• kv: gradual kv95 performance regression since December 1st #62156: gradual kv95 performance regression since December 1st
• YCSB/E performance regression on December 4th #62198: YCSB/E performance regression on December 4th
  • Cause: vectorized execution sql: decrease vectorize_row_count_threshold to 0 #55713
  • Fix: sql: hint scan batch size by expected row count #62282 sql: default to batch size 1 in allocator #62534
  • Backport: release-21.1: sql: hint scan batch size by expected row count #62365 release-21.1: sql: default to batch size 1 in allocator #62603
• storage: investigate YCSB/A regression starting Dec #62523: YCSB/A performance regression on December 23rd
  • Cause: Pebble read compation vendor: Bump pebble to f614b5ad0faa88f794548581b80ad05f99e044ba #58247 options: enable read based compactions pebble#1032
  • Fix: *: Skip first iteration of read sample pebble#1098 vendor: Bump pebble to 294d5706b57a44b72022f93d219c489ebc486e18 #62673
  • Backport: [crl-release-21.1] *: Skip first iteration of read sample pebble#1101

See also investigation spreadsheet.

Branch Benchmarks

kv95/enc=false/nodes=1/cpu=32 on GCE, median cumulative ops/s over 5 runs.

Date	release-20.2	release-21.1	master
2021-03-25	76,018 ops/s	53,462 ops/s	70,149 ops/s
2021-03-29	76,713 ops/s	70,672 ops/s	70,763 ops/s

Follow-up tasks

• cockroachdb/roachperf#49: Add ability to manually annotate graphs
• cockroachdb/roachperf#48: Move roachperf to grafana to improve visualization
• cockroachdb/roachperf#33: capture commit SHA in test metadata
• roachprod: commands for pprof profiling #62309: roachprod: commands for pprof profiling
• roachtest: override (performance) workload flags #62303: roachtest: override workload flags
• roachtest: use smaller workload machines #62302: roachtest: use smaller workload machines
• Write wiki guide for pre-release QA audit of benchmarks and performance regressions

[erikgrinaker]

We've picked all of the low-hanging fruit at this point. We haven't been able to pinpoint the causes of the gradual decline in performance in the Dec-Feb time frame (#62156), the only remaining known regressions are due to separated intents lock table reads (#62078) and tracing. We did a few runs with these disabled:

• release-20.2: 76,713 ops/s
• master: 70,763 ops/s
• w/o lock table reads: 74,707 ops/s
• w/o lock table reads + tracing: 75,422 ops/s

This roughly gets us back to 20.2. I suspect vectorized execution may make up the remainder -- or just an accumulation of other minor regressions.

[erikgrinaker]

I think we've addressed the regressions that we can at this point, and we'll need more targeted optimization efforts to improve hot path performance.

[erikgrinaker]

Did a comparison of 20.2 and 21.1 on all YCSB workloads and a few kv95 ones here: #62078 (comment)