Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: RefreshRange performance regression #98881

Closed
nvanbenschoten opened this issue Mar 17, 2023 · 24 comments · Fixed by #111338
Closed

kvserver: RefreshRange performance regression #98881

nvanbenschoten opened this issue Mar 17, 2023 · 24 comments · Fixed by #111338
Assignees
@jbowens
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-storage Storage Team

Comments

@nvanbenschoten
Copy link
Member

nvanbenschoten commented Mar 17, 2023 •
edited
Loading

See #98068 and https://docs.google.com/spreadsheets/d/10GhYr_91CANCNKOM_gPy7Sk9hQkTyQGNgNwNgfHeUtI/edit#gid=4.

benchdiff --old=81a114c --new=master --post-checkout='dev generate go' --run='BenchmarkRefreshRange' ./pkg/kv/kvserver/batcheval

Jira issue: CRDB-25565
@nvanbenschoten nvanbenschoten added C-performance Perf of queries or internals. Solution not expected to change functional behavior. A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. GA-blocker branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Mar 17, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Mar 17, 2023
@nvanbenschoten nvanbenschoten mentioned this issue Mar 17, 2023
Closed
7 tasks
@nvanbenschoten
Copy link
Member Author 

$ benchdiff --old=81a114c --new=master --post-checkout='dev generate go' --run='BenchmarkRefreshRange' ./pkg/kv/kvserver/batcheval

name                                                      old time/op    new time/op    delta
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-30     135µs ± 1%     130µs ± 1%   -3.93%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-30     135µs ± 1%     130µs ± 1%   -3.86%  (p=0.000 n=9+9)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-30       144µs ± 1%     138µs ± 1%   -3.81%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-30     144µs ± 0%     139µs ± 1%   -3.68%  (p=0.000 n=9+8)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-30     136µs ± 1%     131µs ± 1%   -3.44%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-30       288µs ± 1%     288µs ± 1%     ~     (p=0.912 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-30     553ms ± 0%     617ms ± 1%  +11.63%  (p=0.000 n=9+9)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-30        320µs ± 0%     373µs ± 0%  +16.62%  (p=0.000 n=9+9)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-30    36.7µs ± 1%    43.2µs ± 0%  +17.76%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-30    36.6µs ± 1%    43.3µs ± 1%  +18.11%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-30    40.9µs ± 1%    48.5µs ± 1%  +18.63%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-30    32.4µs ± 1%    39.3µs ± 1%  +21.44%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-30    32.4µs ± 1%    39.4µs ± 1%  +21.67%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-30    32.4µs ± 0%    39.4µs ± 1%  +21.72%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-30    31.0µs ± 1%    38.2µs ± 1%  +23.30%  (p=0.000 n=9+9)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-30     278ms ± 1%     348ms ± 1%  +25.33%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-30      791µs ± 1%     997µs ± 1%  +25.98%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-30     24.2µs ± 1%    31.1µs ± 0%  +28.69%  (p=0.000 n=9+8)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-30     369ms ± 1%     481ms ± 1%  +30.22%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-30     369ms ± 1%     482ms ± 0%  +30.52%  (p=0.000 n=10+8)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-30     19.3µs ± 1%    25.5µs ± 1%  +31.98%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-30     19.3µs ± 1%    25.5µs ± 1%  +32.05%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-30     19.3µs ± 1%    25.5µs ± 1%  +32.06%  (p=0.000 n=9+9)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-30    19.4µs ± 1%    25.7µs ± 1%  +32.20%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-30    25.5µs ± 1%    33.8µs ± 1%  +32.35%  (p=0.000 n=9+9)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-30    19.4µs ± 1%    25.7µs ± 1%  +32.41%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-30      456ms ± 1%     606ms ± 1%  +33.02%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-30     18.8µs ± 1%    25.2µs ± 1%  +33.99%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-30      451ms ± 1%     608ms ± 1%  +34.61%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-30     18.7µs ± 1%    25.2µs ± 1%  +34.64%  (p=0.000 n=9+9)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-30     21.1µs ± 1%    29.3µs ± 0%  +38.79%  (p=0.000 n=9+9)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-30      21.1µs ± 1%    29.4µs ± 1%  +39.11%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-30      21.1µs ± 0%    29.4µs ± 1%  +39.37%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-30     22.8µs ± 1%    31.9µs ± 0%  +40.05%  (p=0.000 n=10+7)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-30     22.8µs ± 1%    31.9µs ± 1%  +40.08%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-30     25.4µs ± 1%    35.6µs ± 1%  +40.16%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-30     21.6µs ± 1%    30.4µs ± 1%  +40.79%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-30     21.6µs ± 1%    30.5µs ± 0%  +41.12%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-30     20.5µs ± 0%    28.9µs ± 1%  +41.15%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-30      22.2µs ± 1%    31.4µs ± 1%  +41.54%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-30      21.0µs ± 1%    29.9µs ± 1%  +42.42%  (p=0.000 n=9+9)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-30     21.0µs ± 1%    31.3µs ± 1%  +48.85%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-30    24.1µs ± 1%    36.0µs ± 1%  +49.55%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-30    24.1µs ± 1%    36.1µs ± 1%  +50.14%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-30      135ms ± 1%     216ms ± 1%  +59.68%  (p=0.000 n=10+10)

name                                                      old alloc/op   new alloc/op   delta
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-30       2.04kB ± 0%    1.67kB ± 0%  -18.14%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-30     2.10kB ± 0%    1.74kB ± 0%  -17.20%  (p=0.000 n=8+10)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-30      4.96kB ± 0%    4.71kB ± 0%   -4.88%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-30    4.96kB ± 0%    4.72kB ± 0%   -4.77%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-30    4.95kB ± 0%    4.72kB ± 0%   -4.74%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-30    4.95kB ± 0%    4.72kB ± 0%   -4.74%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-30    4.95kB ± 0%    4.72kB ± 0%   -4.69%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-30     9.10kB ± 0%    9.81kB ± 0%   +7.83%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-30    9.10kB ± 0%    9.82kB ± 0%   +7.90%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-30      3.49kB ± 0%    3.80kB ± 0%   +8.76%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-30     8.96kB ± 0%    9.77kB ± 0%   +9.08%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-30      8.96kB ± 0%    9.77kB ± 0%   +9.08%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-30     8.95kB ± 0%    9.77kB ± 0%   +9.17%  (p=0.000 n=8+8)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-30    11.3kB ± 0%    12.4kB ± 0%   +9.22%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-30      8.85kB ± 0%    9.66kB ± 0%   +9.23%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-30    11.3kB ± 0%    12.4kB ± 0%   +9.24%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-30     8.85kB ± 0%    9.67kB ± 0%   +9.25%  (p=0.000 n=10+8)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-30     8.85kB ± 0%    9.67kB ± 0%   +9.27%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-30    12.0kB ± 0%    13.1kB ± 0%   +9.52%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-30      9.12kB ± 0%   10.02kB ± 0%   +9.87%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-30      9.12kB ± 0%   10.02kB ± 0%   +9.88%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-30     9.12kB ± 0%   10.02kB ± 0%   +9.90%  (p=0.000 n=8+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-30    10.5kB ± 0%    11.5kB ± 0%  +10.00%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-30    10.5kB ± 0%    11.5kB ± 0%  +10.03%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-30     9.01kB ± 0%    9.92kB ± 0%  +10.05%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-30    10.4kB ± 0%    11.5kB ± 0%  +10.06%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-30    71.4kB ± 4%    78.8kB ±14%  +10.34%  (p=0.016 n=8+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-30    10.1kB ± 0%    11.2kB ± 0%  +10.39%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-30     8.65kB ± 0%    9.61kB ± 0%  +11.03%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-30    8.65kB ± 0%    9.61kB ± 0%  +11.04%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-30    8.65kB ± 0%    9.61kB ± 0%  +11.07%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-30     9.46kB ± 0%   10.51kB ± 0%  +11.09%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-30     8.65kB ± 0%    9.61kB ± 0%  +11.10%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-30     8.65kB ± 0%    9.61kB ± 0%  +11.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-30     8.65kB ± 0%    9.61kB ± 0%  +11.12%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-30     8.65kB ± 0%    9.61kB ± 0%  +11.13%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-30    68.9kB ±13%    77.2kB ±12%  +12.03%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-30     9.10kB ± 0%   10.59kB ± 0%  +16.38%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-30    91.4kB ± 9%   107.4kB ± 9%  +17.42%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-30    9.38kB ± 0%   11.21kB ± 0%  +19.48%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-30    9.37kB ± 0%   11.20kB ± 0%  +19.50%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-30     13.0kB ± 0%    20.1kB ± 1%  +55.54%  (p=0.000 n=8+9)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-30    49.1kB ±11%    76.4kB ± 5%  +55.71%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-30     26.9kB ±20%    44.5kB ±21%  +65.33%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-30     29.1kB ± 1%    48.4kB ± 1%  +66.23%  (p=0.000 n=8+9)

name                                                      old allocs/op  new allocs/op  delta
RefreshRange/random-keys/refresh_window=[95.00,95.00]-30       865 ± 2%       824 ±23%     ~     (p=0.195 n=8+10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-30       804 ±20%       800 ±20%     ~     (p=0.383 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-30        96.0 ± 0%      98.0 ± 0%   +2.08%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-30      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-30      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-30      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-30      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-30       931 ±11%      1026 ±10%  +10.23%  (p=0.023 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-30       27.0 ± 0%      30.0 ± 0%  +11.11%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-30         26.0 ± 0%      29.0 ± 0%  +11.54%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-30       133 ± 0%       154 ± 0%  +15.79%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-30       118 ± 0%       137 ± 0%  +16.10%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-30       118 ± 0%       137 ± 0%  +16.10%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-30       100 ± 0%       119 ± 0%  +19.00%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-30       100 ± 0%       119 ± 0%  +19.00%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-30       100 ± 0%       119 ± 0%  +19.00%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-30       69.0 ± 0%      83.0 ± 0%  +20.29%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-30      69.0 ± 0%      83.0 ± 0%  +20.29%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-30      92.0 ± 0%     113.0 ± 0%  +22.83%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-30        63.0 ± 0%      78.0 ± 0%  +23.81%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-30       77.0 ± 0%      97.0 ± 0%  +25.97%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-30        63.0 ± 0%      82.0 ± 0%  +30.16%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-30       63.0 ± 0%      82.0 ± 0%  +30.16%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-30       63.0 ± 0%      82.0 ± 0%  +30.16%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-30       59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-30       59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-30      59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-30      59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-30       59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-30       59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-30       59.0 ± 0%      77.0 ± 0%  +30.51%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-30        65.0 ± 0%      85.0 ± 0%  +30.77%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-30        65.0 ± 0%      85.0 ± 0%  +30.77%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-30       65.0 ± 0%      85.0 ± 0%  +30.77%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-30        60.0 ± 0%      79.0 ± 0%  +31.67%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-30       60.0 ± 0%      79.0 ± 0%  +31.67%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-30       60.0 ± 0%      79.0 ± 0%  +31.67%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-30       62.0 ± 0%      83.0 ± 0%  +33.87%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-30        108 ± 0%       150 ± 1%  +38.66%  (p=0.000 n=8+9)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-30       537 ±15%       746 ± 7%  +38.91%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-30       69.0 ± 0%      99.0 ± 0%  +43.48%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-30      76.0 ± 0%     113.0 ± 0%  +48.68%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-30      76.0 ± 0%     113.0 ± 0%  +48.68%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-30        177 ±26%       306 ±33%  +72.77%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-30        195 ± 2%       349 ± 2%  +79.15%  (p=0.000 n=8+9)

@andrewbaptist
Copy link
Collaborator

I think I tracked down the cause of the perf regression for BenchmarkRefreshRange and it seems to be in the conversion of error messages to strings as a result of PCI compliance:
https://cockroachlabs.atlassian.net/browse/CRDB-14087
3e1cdb0 - I’m not 100% sure yet, but here are the flame graphs for a 10s run:

The big difference in the time to format the error as part of the regex here:

cockroach/pkg/kv/kvserver/batcheval/cmd_refresh_range_bench_test.go

Line 160 in 736a67e

require.Regexp(b, "encountered recently written committed value", err)

Here are what the old and new flame graphs look like (notice the big redact.Sprintf section. So the good news is that I don't think we "broke" anything related to range refreshes, the bad news is that this could affect a lot of tests. I'm going to look at rewriting the test to not include the assertion time.
Old:
image

New:
image

@andrewbaptist
Copy link
Collaborator

After running with the patched version (to remove the expensive assert) the new version still looks worse:

Found previous run; old=benchdiff/fda44ff/artifacts/out.2023-03-24T19_45_27Z, new=benchdiff/f48925d/artifacts/out.2023-03-24T19_45_27Z
name                                                      old time/op    new time/op    delta
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24     149µs ± 0%     141µs ± 1%   -5.09%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24     158µs ± 1%     150µs ± 1%   -5.04%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24     148µs ± 0%     141µs ± 1%   -4.80%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24       158µs ± 1%     151µs ± 1%   -4.59%  (p=0.000 n=9+10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24     149µs ± 0%     143µs ± 1%   -3.93%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24       319µs ± 1%     317µs ± 1%   -0.67%  (p=0.002 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24    28.6µs ± 1%    29.2µs ± 1%   +1.85%  (p=0.000 n=10+8)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24    28.6µs ± 1%    29.2µs ± 1%   +2.02%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24    24.5µs ± 1%    25.1µs ± 1%   +2.40%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24    24.4µs ± 0%    25.0µs ± 1%   +2.52%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24    24.5µs ± 1%    25.1µs ± 1%   +2.71%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24    32.9µs ± 1%    33.9µs ± 1%   +2.98%  (p=0.000 n=9+9)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24     11.5µs ± 0%    12.1µs ± 2%   +5.63%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24    11.5µs ± 1%    12.2µs ± 2%   +5.67%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24     11.0µs ± 1%    11.6µs ± 1%   +5.80%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24     11.4µs ± 1%    12.1µs ± 1%   +5.80%  (p=0.000 n=8+10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24    11.5µs ± 0%    12.2µs ± 1%   +5.99%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24    23.1µs ± 1%    24.5µs ± 1%   +6.03%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24     11.0µs ± 1%    11.7µs ± 2%   +6.39%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24     11.4µs ± 1%    12.2µs ± 2%   +6.68%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24     16.2µs ± 1%    17.5µs ± 1%   +8.07%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     614ms ± 1%     686ms ± 1%  +11.63%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24    17.8µs ± 1%    20.7µs ± 2%  +16.50%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24        356µs ± 1%     414µs ± 1%  +16.52%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24      13.2µs ± 1%    16.1µs ± 1%  +21.86%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24     13.2µs ± 1%    16.1µs ± 1%  +22.03%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24      13.2µs ± 0%    16.1µs ± 1%  +22.06%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24     12.6µs ± 1%    15.4µs ± 1%  +22.30%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24      883µs ± 1%    1094µs ± 0%  +23.90%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24     310ms ± 0%     386ms ± 1%  +24.39%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24     13.9µs ± 1%    17.4µs ± 1%  +25.14%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24     13.9µs ± 1%    17.4µs ± 2%  +25.30%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24     15.0µs ± 2%    18.8µs ± 1%  +25.42%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24     15.0µs ± 1%    18.9µs ± 2%  +25.45%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24      13.4µs ± 1%    16.9µs ± 3%  +26.53%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24      14.5µs ± 1%    18.3µs ± 2%  +26.76%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24     17.7µs ± 1%    22.8µs ± 1%  +28.83%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24     406ms ± 1%     534ms ± 1%  +31.64%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24     408ms ± 1%     538ms ± 2%  +32.02%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24      509ms ± 1%     675ms ± 1%  +32.42%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24      505ms ± 1%     674ms ± 1%  +33.48%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24     13.2µs ± 1%    17.7µs ± 1%  +34.15%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24    16.1µs ± 1%    22.1µs ± 2%  +37.09%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24    16.1µs ± 1%    22.2µs ± 2%  +37.52%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24      148ms ± 1%     239ms ± 2%  +61.73%  (p=0.000 n=9+10)

name                                                      old alloc/op   new alloc/op   delta
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24     2.07kB ± 0%    1.50kB ± 0%  -27.47%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24    2.07kB ± 0%    1.50kB ± 0%  -27.45%  (p=0.000 n=7+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24      1.82kB ± 0%    1.35kB ± 0%  -25.55%  (p=0.002 n=8+10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24     1.82kB ± 0%    1.35kB ± 0%  -25.55%  (p=0.000 n=8+8)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24     1.82kB ± 0%    1.35kB ± 0%  -25.55%  (p=0.000 n=9+7)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24      1.92kB ± 0%    1.46kB ± 0%  -24.15%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24     1.92kB ± 0%    1.46kB ± 0%  -24.15%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24     1.92kB ± 0%    1.46kB ± 0%  -24.14%  (p=0.000 n=8+10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24     1.62kB ± 0%    1.29kB ± 0%  -20.29%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24    1.62kB ± 0%    1.29kB ± 0%  -20.28%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24     1.62kB ± 0%    1.29kB ± 0%  -20.28%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24     1.62kB ± 0%    1.29kB ± 0%  -20.28%  (p=0.000 n=10+8)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24    1.62kB ± 0%    1.29kB ± 0%  -20.27%  (p=0.000 n=10+8)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24     1.62kB ± 0%    1.29kB ± 0%  -20.27%  (p=0.000 n=10+8)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24     1.62kB ± 0%    1.29kB ± 0%  -20.26%  (p=0.000 n=10+9)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24     1.98kB ± 0%    1.60kB ± 0%  -19.03%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24      2.08kB ± 0%    1.70kB ± 0%  -18.48%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24     2.08kB ± 0%    1.70kB ± 0%  -18.47%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24      2.08kB ± 0%    1.70kB ± 0%  -18.45%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24       2.04kB ± 0%    1.67kB ± 0%  -18.00%  (p=0.000 n=10+8)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24     2.10kB ± 0%    1.75kB ± 0%  -16.67%  (p=0.000 n=10+8)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24     2.42kB ± 0%    2.18kB ± 0%   -9.89%  (p=0.000 n=10+7)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24    3.40kB ± 0%    3.08kB ± 0%   -9.43%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24    3.40kB ± 0%    3.08kB ± 0%   -9.43%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24    3.40kB ± 0%    3.08kB ± 0%   -9.42%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24    4.26kB ± 0%    3.94kB ± 0%   -7.54%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24    4.26kB ± 0%    3.94kB ± 0%   -7.53%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24    3.11kB ± 0%    2.88kB ± 0%   -7.44%  (p=0.000 n=10+7)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24      4.95kB ± 0%    4.71kB ± 0%   -4.85%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24    4.95kB ± 0%    4.72kB ± 0%   -4.77%  (p=0.000 n=10+9)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24    4.90kB ± 0%    4.66kB ± 0%   -4.76%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24    4.95kB ± 0%    4.72kB ± 0%   -4.70%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24    4.95kB ± 0%    4.72kB ± 0%   -4.69%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24    4.95kB ± 0%    4.72kB ± 0%   -4.67%  (p=0.000 n=7+10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24      3.49kB ± 0%    3.80kB ± 0%   +8.76%  (p=0.000 n=10+9)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24     2.07kB ± 0%    2.27kB ± 0%   +9.71%  (p=0.001 n=7+7)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24    93.1kB ± 4%   104.2kB ± 9%  +11.89%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24     38.1kB ± 0%    43.7kB ± 1%  +14.86%  (p=0.000 n=8+9)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24    2.34kB ± 0%    2.88kB ± 0%  +22.97%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24    2.34kB ± 0%    2.88kB ± 0%  +22.98%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24     13.0kB ± 2%    16.9kB ±21%  +30.06%  (p=0.000 n=8+10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24     33.5kB ±20%    43.7kB ± 1%  +30.45%  (p=0.000 n=10+8)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24    49.4kB ± 4%    72.4kB ± 8%  +46.53%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24    70.5kB ± 4%   110.1kB ±14%  +56.16%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24    70.9kB ± 4%   111.2kB ±10%  +56.80%  (p=0.000 n=8+10)

name                                                      old allocs/op  new allocs/op  delta
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24       32.0 ± 0%      28.0 ± 0%  -12.50%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24      32.0 ± 0%      28.0 ± 0%  -12.50%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24       22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24       22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24       22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24       22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24       22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     1.01k ± 3%     1.05k ±11%     ~     (p=0.072 n=10+10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24      22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24      22.0 ± 0%      22.0 ± 0%     ~     (all equal)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24        113 ± 3%       137 ±26%     ~     (p=0.162 n=8+10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24      81.0 ± 0%      82.0 ± 0%   +1.23%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24      81.0 ± 0%      82.0 ± 0%   +1.23%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24      63.0 ± 0%      64.0 ± 0%   +1.59%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24      63.0 ± 0%      64.0 ± 0%   +1.59%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24      63.0 ± 0%      64.0 ± 0%   +1.59%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24        96.0 ± 0%      98.0 ± 0%   +2.08%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24      96.0 ± 0%      99.0 ± 0%   +3.12%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24        26.0 ± 0%      27.0 ± 0%   +3.85%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24       26.0 ± 0%      27.0 ± 0%   +3.85%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24       26.0 ± 0%      27.0 ± 0%   +3.85%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24        23.0 ± 0%      24.0 ± 0%   +4.35%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24       23.0 ± 0%      24.0 ± 0%   +4.35%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24       23.0 ± 0%      24.0 ± 0%   +4.35%  (p=0.000 n=10+10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24       40.0 ± 0%      42.0 ± 0%   +5.00%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24      55.0 ± 0%      58.0 ± 0%   +5.45%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24        28.0 ± 0%      30.0 ± 0%   +7.14%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24        28.0 ± 0%      30.0 ± 0%   +7.14%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24       28.0 ± 0%      30.0 ± 0%   +7.14%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24       27.0 ± 0%      30.0 ± 0%  +11.11%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24         26.0 ± 0%      29.0 ± 0%  +11.54%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24       25.0 ± 0%      28.0 ± 0%  +12.00%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24        63.0 ± 0%      78.0 ± 0%  +23.81%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24       578 ± 4%       737 ±10%  +27.42%  (p=0.000 n=9+10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24        268 ± 1%       349 ± 1%  +30.27%  (p=0.000 n=8+9)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24       880 ± 3%      1196 ±22%  +35.81%  (p=0.000 n=9+10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24       32.0 ± 0%      44.0 ± 0%  +37.50%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24       883 ± 3%      1293 ± 2%  +46.53%  (p=0.000 n=8+8)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24      39.0 ± 0%      58.0 ± 0%  +48.72%  (p=0.000 n=10+10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24      39.0 ± 0%      58.0 ± 0%  +48.72%  (p=0.000 n=10+10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24        230 ±24%       349 ± 2%  +51.61%  (p=0.000 n=10+8)

More investigation needed...

@andrewbaptist andrewbaptist mentioned this issue Mar 24, 2023
Merged
@andrewbaptist
Copy link
Collaborator

The difference between the two runs now is pretty small, however there are certainly some differences. I normalized the runs based on the number of iterations. There is no "big ticket" item that is difference, however in many of the calls there is some extra work done. Here is an example of the close:
Old:
image
New:
image
Notice there is a small extra 89ms/op for closing objstorage that wasn't there before.

This doesn't account for the full difference however so there is likely more calls to the underlying iterator.

@nvanbenschoten
Copy link
Member Author

Notice there is a small extra 89ms/op for closing objstorage that wasn't there before.

This is interesting. We're not using object storage, so it's worth reaching out to Storage to see whether they expect this sync.Pool access to be prominent in CPU profiles. Can we avoid touching the sync.Pool when object storage is not in use?

andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Mar 29, 2023
@andrewbaptist
kv: remove expensive print
e5e47ab 
Removes an unnecessarily expensive assertion from the bench test as it
was taking >50% of the runtime. Now the test correctly spends time on
the important code.

Informs: cockroachdb#98881

Epic: none
Release note: None
craig bot pushed a commit that referenced this issue Mar 30, 2023
@andrewbaptist
Merge #99520
56e890f 
99520: kv: remove expensive print r=nvanbenschoten a=andrewbaptist

Removes an unnecessarily expensive assertion from the bench test as it was taking >50% of the runtime. Now the test correctly spends time on the important code.

Informs: #98881

Epic: none
Release note: None

Co-authored-by: Andrew Baptist <[email protected]>
@blathers-crl blathers-crl bot mentioned this issue Apr 2, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Apr 2, 2023
@andrewbaptist
kv: remove expensive print
24d531c 
Removes an unnecessarily expensive assertion from the bench test as it
was taking >50% of the runtime. Now the test correctly spends time on
the important code.

Informs: #98881

Epic: none
Release note: None
@jbowens
Copy link
Collaborator

jbowens commented Apr 17, 2023 •
edited
Loading

Thinking about it more, I think all those observations jive with the theory of value blocks increasing the cost of moving to the next visible key. Time-bound iteration happens at the ssblock level at its most granular, so the denser ssblocks cause more out-of-timerange keys to be surfaced up to the MVCC layer. cockroachdb/pebble#1446 describes a related idea that would perform the time-bound filtering at the leaf, avoiding the overhead of bubbling these out-of-bound keys up.

@sumeerbhola does this make sense to you? I don't think there's reason to change anything about this release, but maybe there's an argument to revisit cockroachdb/pebble#1446 as value blocks have increased its significance.

I'm going to disable value blocks and collect some numbers to get confirmation.

@jbowens
Copy link
Collaborator

jbowens commented Apr 17, 2023

Dang, this was a miss. Without value blocks, there's only a ~5% improvement on the most regressed RefreshRange/mixed-case/refresh_window=[99.00,99.00] benchmark variant.

                                                         │   sec/op    │   sec/op     vs base              │
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24     175.6µ ± 1%   178.2µ ± 2%       ~ (p=0.180 n=6)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24    17.58µ ± 1%   17.69µ ± 1%       ~ (p=0.065 n=6)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24    15.48µ ± 1%   15.47µ ± 2%       ~ (p=0.937 n=6)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24    15.37µ ± 1%   15.35µ ± 1%       ~ (p=0.394 n=6)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24    15.46µ ± 1%   15.49µ ± 2%       ~ (p=0.485 n=6)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24   174.5µ ± 1%   172.0µ ± 4%       ~ (p=0.065 n=6)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24   22.77µ ± 1%   22.53µ ± 1%  -1.07% (p=0.026 n=6)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24   22.76µ ± 1%   22.75µ ± 2%       ~ (p=0.818 n=6)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24   22.77µ ± 0%   22.80µ ± 1%       ~ (p=0.240 n=6)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24   163.1µ ± 1%   162.8µ ± 4%       ~ (p=0.699 n=6)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24   24.10µ ± 2%   24.35µ ± 1%       ~ (p=0.102 n=6)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24   24.14µ ± 1%   24.29µ ± 2%       ~ (p=0.240 n=6)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24   162.9µ ± 1%   161.4µ ± 1%  -0.88% (p=0.002 n=6)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24   26.73µ ± 0%   26.81µ ± 1%       ~ (p=0.589 n=6)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24   166.9µ ± 1%   164.5µ ± 3%       ~ (p=0.132 n=6)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24     390.7µ ± 1%   384.0µ ± 1%  -1.71% (p=0.002 n=6)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24    17.46µ ± 1%   17.48µ ± 2%       ~ (p=1.000 n=6)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24    19.28µ ± 1%   19.04µ ± 2%  -1.24% (p=0.041 n=6)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24    14.91µ ± 1%   14.61µ ± 2%  -2.01% (p=0.041 n=6)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24    14.84µ ± 1%   14.72µ ± 2%       ~ (p=0.310 n=6)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24   871.1m ± 2%   853.8m ± 2%  -1.99% (p=0.002 n=6)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24   25.03µ ± 1%   25.09µ ± 1%       ~ (p=0.240 n=6)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24   15.51µ ± 1%   15.49µ ± 1%       ~ (p=0.699 n=6)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24   15.37µ ± 1%   15.46µ ± 2%       ~ (p=0.394 n=6)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24   489.9m ± 1%   479.2m ± 2%  -2.18% (p=0.002 n=6)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24   20.53µ ± 1%   20.46µ ± 1%       ~ (p=0.485 n=6)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24   20.44µ ± 0%   20.65µ ± 1%  +0.99% (p=0.015 n=6)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24   679.5m ± 1%   657.8m ± 2%  -3.20% (p=0.002 n=6)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24   23.48µ ± 0%   23.21µ ± 2%  -1.14% (p=0.026 n=6)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24   683.8m ± 1%   658.6m ± 3%  -3.68% (p=0.004 n=6)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24      530.6µ ± 2%   514.7µ ± 1%  -2.99% (p=0.002 n=6)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24     18.57µ ± 4%   18.42µ ± 1%  -0.81% (p=0.015 n=6)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24     18.66µ ± 1%   18.50µ ± 1%       ~ (p=0.132 n=6)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24     22.75µ ± 0%   22.56µ ± 2%       ~ (p=0.240 n=6)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24     21.82µ ± 3%   21.24µ ± 1%  -2.65% (p=0.002 n=6)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24    858.7m ± 1%   831.7m ± 4%       ~ (p=0.310 n=6)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24    27.02µ ± 0%   26.76µ ± 2%  -0.96% (p=0.004 n=6)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24    23.44µ ± 2%   23.35µ ± 2%       ~ (p=0.310 n=6)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24    22.32µ ± 1%   22.10µ ± 2%       ~ (p=0.132 n=6)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24    857.2m ± 1%   839.6m ± 2%  -2.06% (p=0.002 n=6)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24    23.30µ ± 1%   23.01µ ± 3%       ~ (p=0.065 n=6)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24    22.29µ ± 1%   21.91µ ± 3%       ~ (p=0.394 n=6)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24    1.389m ± 1%   1.362m ± 2%  -1.97% (p=0.002 n=6)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24    18.23µ ± 1%   18.22µ ± 1%       ~ (p=0.818 n=6)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24    306.0m ± 3%   291.5m ± 4%  -4.73% (p=0.004 n=6)

@sumeerbhola
Copy link
Collaborator

@sumeerbhola does this make sense to you?

Makes sense

Diffing and normalizing the profiles ...

What exactly did you do here -- I want to make sure I look at the same thing.

@jbowens
Copy link
Collaborator

jbowens commented Apr 18, 2023 

go tool pprof --normalize -diff_base ~/Downloads/cpu-22.2.pprof.gz ~/Downloads/cpu-master.pprof.gz

using Erik's profiles

@jbowens
Copy link
Collaborator

jbowens commented Apr 18, 2023

I took some new profiles of just the benchmark variant with the largest regression RefreshRange/mixed-case/refresh_window=[99.00,99.00]:

Screenshot 2023-04-18 at 10 15 23 AM

master.cpu.pprof.gz

Screenshot 2023-04-18 at 10 15 06 AM

81a114c.cpu.pprof.gz

go tool pprof --http :6060 --normalize -focus='batcheval\.RefreshRange' --diff_base 81a114c.cpu.pprof.gz master.cpu.pprof.gz

This seems to show most of the additional time being spent in the maybeSkipKeys instead. I think those broader cpu profiles were confounding, because the other benchmarks are now faster and complete more runs relative to the slower, still regressed microbenchmark variants.

@jbowens
Copy link
Collaborator

jbowens commented Apr 18, 2023

It does look like we're losing both the monotonic bounds optimization and the TrySeekUsingNext optimization:

         .          .    769:	var dontSeekWithinBlock bool
   17.08ms    85.39ms    770:	if !i.data.isDataInvalidated() && !i.index.isDataInvalidated() && i.data.valid() && i.index.valid() &&
         .          .    771:		boundsCmp > 0 && i.cmp(key, i.index.Key().UserKey) <= 0 {
         .          .    772:		// Fast-path: The bounds have moved forward and this SeekGE is
         .          .    773:		// respecting the lower bound (guaranteed by Iterator). We know that
         .          .    774:		// the iterator must already be positioned within or just outside the
         .          .    775:		// previous bounds. Therefore it cannot be positioned at a block (or
         .          .    776:		// the position within that block) that is ahead of the seek position.
         .          .    777:		// However it can be positioned at an earlier block. This fast-path to
         .          .    778:		// use Next() on the block is only applied when we are already at the
         .          .    779:		// block that the slow-path (the else-clause) would load -- this is
         .          .    780:		// the motivation for the i.cmp(key, i.index.Key().UserKey) <= 0
         .          .    781:		// predicate.
         .          .    782:		i.initBoundsForAlreadyLoadedBlock()
     -30ms     -7.29s    783:		ikey, val, done := i.trySeekGEUsingNextWithinBlock(key)
         .          .    784:		if done {
         .          .    785:			return ikey, val
         .          .    786:		}
         .          .    787:		if ikey == nil {
         .          .    788:			// Done with this block.
         .          .    789:			dontSeekWithinBlock = true
     -20ms    -11.60s    790:		}
         .          .    791:	} else {
         .          .    792:		// Cannot use bounds monotonicity. But may be able to optimize if
         .          .    793:		// caller claimed externally known invariant represented by
         .          .    794:		// flags.TrySeekUsingNext().
         .          .    795:		if flags.TrySeekUsingNext() {
         .          .    796:			// seekPrefixGE or SeekGE has already ensured
         .          .    797:			// !i.data.isDataInvalidated() && i.exhaustedBounds != +1
         .          .    798:			currKey := i.data.Key()
         .          .    799:			value := i.data.value()
         .          .    800:			less := i.cmp(currKey.UserKey, key) < 0
         .          .    801:			// We could be more sophisticated and confirm that the seek
         .          .    802:			// position is within the current block before applying this
         .          .    803:			// optimization. But there may be some benefit even if it is in
         .          .    804:			// the next block, since we can avoid seeking i.index.
         .          .    805:			for j := 0; less && j < numStepsBeforeSeek; j++ {
         .          .    806:				currKey, value = i.Next()
         .          .    807:				if currKey == nil {
     -10ms      -10ms    808:					return nil, base.LazyValue{}
     -20ms    -11.31s    809:				}
     -40ms      -40ms    810:				less = i.cmp(currKey.UserKey, key) < 0
         .          .    811:			}
         .          .    812:			if !less {
         .          .    813:				if i.blockUpper != nil && i.cmp(currKey.UserKey, i.blockUpper) >= 0 {
         .          .    814:					i.exhaustedBounds = +1
         .          .    815:					return nil, base.LazyValue{}
         .          .    816:				}
     -20ms     -560ms    817:				return currKey, value
         .          .    818:			}
         .          .    819:		}

and then the slow path:

         .          .    821:		// Slow-path.
         .          .    822:
         .          .    823:		// Since we're re-seeking the iterator, the previous value of
         .          .    824:		// maybeFilteredKeysSingleLevel is irrelevant. If we filter out blocks
         .          .    825:		// during seeking, loadBlock will set it to true.
         .          .    826:		i.maybeFilteredKeysSingleLevel = false
         .          .    827:
         .          .    828:		var ikey *InternalKey
   17.08ms     12.01s    829:		if ikey, _ = i.index.SeekGE(key, flags.DisableTrySeekUsingNext()); ikey == nil {
         .          .    830:			// The target key is greater than any key in the index block.
         .          .    831:			// Invalidate the block iterator so that a subsequent call to Prev()
         .          .    832:			// will return the last key in the table.
         .          .    833:			i.data.invalidate()
         .          .    834:			return nil, base.LazyValue{}
         .          .    835:		}
   34.16ms     30.79s    836:		result := i.loadBlock(+1)
   17.08ms    17.08ms    837:		if result == loadBlockFailed {
         .          .    838:			return nil, base.LazyValue{}
         .          .    839:		}
         .          .    840:		if result == loadBlockIrrelevant {
         .          .    841:			// Enforce the upper bound here since don't want to bother moving
         .          .    842:			// to the next block if upper bound is already exceeded. Note that
         .          .    843:			// the next block starts with keys >= ikey.UserKey since even
         .          .    844:			// though this is the block separator, the same user key can span
         .          .    845:			// multiple blocks. Since upper is exclusive we use >= below.
         .          .    846:			if i.upper != nil && i.cmp(ikey.UserKey, i.upper) >= 0 {
         .          .    847:				i.exhaustedBounds = +1
         .          .    848:				return nil, base.LazyValue{}
         .          .    849:			}
         .          .    850:			// Want to skip to the next block.
         .          .    851:			dontSeekWithinBlock = true
         .          .    852:		}
         .          .    853:	}
   17.08ms    17.08ms    854:	if !dontSeekWithinBlock {
   34.16ms     19.67s    855:		if ikey, val := i.data.SeekGE(key, flags.DisableTrySeekUsingNext()); ikey != nil {
   68.31ms    68.31ms    856:			if i.blockUpper != nil && i.cmp(ikey.UserKey, i.blockUpper) >= 0 {
         .          .    857:				i.exhaustedBounds = +1
         .          .    858:				return nil, base.LazyValue{}
         .          .    859:			}
   34.16ms    34.16ms    860:			return ikey, val
         .          .    861:		}
         .          .    862:	}
   68.31ms      1.35s    863:	return i.skipForward()
         .          .    864:}

@jbowens
Copy link
Collaborator

jbowens commented Apr 19, 2023 •
edited
Loading

Looks like there's some variance in this benchmark, and this is value blocks. It's not the case that we're losing the try-seek-using-next optimization. We almost never had it to begin with. The negative duration lines were actually also the slow path, just different line numbers on the old Pebble SHA. They just happened to align with the fast path code block on the new SHA.

We almost never hit the TrySeekUsingNext optimization because in the typical case we Next/NextKey the main iterator roughly in tandem with the time-bound iterator. This has been the case since #45785.

There are a few mitigations that we could apply:

  1. @sumeerbhola suggested cutting to the next ssblock early if we've written a certain density of keys to a block.
  2. db: push down time-bound iterator filtering pebble#1446 would avoid propagating keys outside the filter beyond the block iterator.
  3. A new incremental iterator implementation that avoids the dual iterator approach would also help. storage: replacement for MVCCIncrementalIterator for separated intents #66869

I don't think this needs to block GA. Removing the label.

@RaduBerinde RaduBerinde removed their assignment Sep 14, 2023
@jbowens jbowens mentioned this issue Sep 27, 2023
Merged
jbowens added a commit to jbowens/cockroach that referenced this issue Sep 27, 2023
@jbowens
storage: strengthen time-bound iteration guarantees
69f3629 
Previously, the minimum and maximum timestamps used to configure a "time-bound"
iterator were only hints. The iterator would surface keys outside those bounds
frequently if the keys were contained within a sstable block that also
contained keys within bounds. This commit adapts the iterator construction to
use Pebble's IterOptions.SkipPoint option to filter out-of-range keys without
yielding them to the caller.

This adds a small additional overhead to processing KVs that are ultimately
yielded to the user of a MVCCIncrementalIterator, but removes a large overhead
to proessing KVs that are ultimately skipped. In the current
MVCCIncrementalIterator implementation, keys that are outside the time range
still reposition the iterator without time bound. This repositioning no longer
happens, and removing this overhead more than reclaims the RefreshRange
benchmark's regression from the introduction value blocks packing keys more
densely.

This also serves as a step towards cockroachdb#66869.

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                         │    old.txt    │               new.txt               │
                                                         │    sec/op     │   sec/op     vs base                │
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24      150.73µ ± 1%   55.75µ ± 1%  -63.01% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24      16.52µ ± 1%   16.19µ ± 2%   -2.00% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24      13.94µ ± 1%   13.59µ ± 1%   -2.55% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24      13.97µ ± 1%   13.56µ ± 3%   -2.99% (p=0.002 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24      13.96µ ± 1%   13.45µ ± 3%   -3.69% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24    149.95µ ± 1%   55.67µ ± 1%  -62.88% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24     22.05µ ± 1%   21.70µ ± 0%   -1.57% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24     22.08µ ± 1%   21.61µ ± 0%   -2.14% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24     21.91µ ± 1%   21.69µ ± 1%   -0.99% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24    139.27µ ± 1%   55.82µ ± 1%  -59.92% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24     23.76µ ± 1%   23.36µ ± 0%   -1.66% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24     23.85µ ± 1%   23.25µ ± 1%   -2.48% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24    138.94µ ± 1%   55.52µ ± 0%  -60.04% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24     26.68µ ± 1%   26.30µ ± 1%   -1.42% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24    141.39µ ± 1%   55.02µ ± 1%  -61.09% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24      322.35µ ± 0%   51.21µ ± 0%  -84.11% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24      16.35µ ± 1%   16.63µ ± 2%        ~ (p=0.105 n=10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24      17.42µ ± 1%   17.29µ ± 3%        ~ (p=0.796 n=10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24      13.40µ ± 1%   13.45µ ± 1%        ~ (p=0.529 n=10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24      13.59µ ± 2%   13.41µ ± 0%        ~ (p=0.239 n=10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     707.5m ± 1%   111.7m ± 1%  -84.21% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24     21.94µ ± 1%   17.86µ ± 0%  -18.59% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24     13.95µ ± 1%   13.54µ ± 1%   -2.94% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24     13.96µ ± 0%   13.53µ ± 1%   -3.06% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24    395.22m ± 0%   33.81m ± 1%  -91.45% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24     19.42µ ± 1%   19.40µ ± 1%        ~ (p=0.912 n=10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24     19.37µ ± 1%   19.42µ ± 1%        ~ (p=0.247 n=10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24    546.66m ± 3%   48.48m ± 3%  -91.13% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24     21.69µ ± 1%   19.68µ ± 1%   -9.24% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24    546.27m ± 0%   48.73m ± 3%  -91.08% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24        410.1µ ± 0%   221.1µ ± 1%  -46.09% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24       14.84µ ± 1%   14.84µ ± 1%        ~ (p=0.739 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24       14.84µ ± 1%   14.83µ ± 1%        ~ (p=0.615 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24       17.39µ ± 2%   17.51µ ± 2%   +0.68% (p=0.029 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24       16.41µ ± 1%   16.38µ ± 1%        ~ (p=0.971 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24      660.9m ± 0%   140.6m ± 3%  -78.73% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24      20.43µ ± 1%   15.46µ ± 1%  -24.31% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24      18.03µ ± 1%   17.66µ ± 1%   -2.01% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24      16.86µ ± 1%   16.70µ ± 1%   -0.93% (p=0.011 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24      659.5m ± 1%   137.1m ± 3%  -79.21% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24      17.96µ ± 1%   17.65µ ± 1%   -1.77% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24      16.86µ ± 1%   16.66µ ± 1%   -1.23% (p=0.008 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24     900.95µ ± 0%   32.18µ ± 0%  -96.43% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24      14.35µ ± 1%   14.50µ ± 2%   +1.03% (p=0.002 n=10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24    227.235m ± 1%   3.361m ± 1%  -98.52% (p=0.000 n=10)
geomean                                                      136.9µ        73.57µ       -46.24%

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                    │  old2.txt   │              new2.txt               │
                                                    │   sec/op    │   sec/op     vs base                │
MVCCIncrementalIterator/ts=5-24                       24.93m ± 3%   24.53m ± 2%   -1.61% (p=0.004 n=10)
MVCCIncrementalIterator/ts=480-24                     560.4µ ± 1%   453.8µ ± 0%  -19.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=100-24    1.444m ± 0%   1.434m ± 0%   -0.72% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=500-24    1.994m ± 0%   1.974m ± 1%   -1.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=1000-24   2.672m ± 1%   2.649m ± 1%        ~ (p=0.063 n=10)
MVCCIncrementalIteratorForOldData/valueSize=2000-24   4.059m ± 1%   4.009m ± 1%   -1.23% (p=0.011 n=10)
geomean                                               2.754m        2.635m        -4.33%
```

Epic: none
Informs cockroachdb#66869
Close cockroachdb#98881
Release note: none
jbowens added a commit to jbowens/cockroach that referenced this issue Sep 29, 2023
@jbowens
storage: strengthen time-bound iteration guarantees
507d2b0 
Previously, the minimum and maximum timestamps used to configure a "time-bound"
iterator were only hints. The iterator would surface keys outside those bounds
frequently if the keys were contained within a sstable block that also
contained keys within bounds. This commit adapts the iterator construction to
use Pebble's IterOptions.SkipPoint option to filter out-of-range keys without
yielding them to the caller.

This adds a small additional overhead to processing KVs that are ultimately
yielded to the user of a MVCCIncrementalIterator, but removes a large overhead
to proessing KVs that are ultimately skipped. In the current
MVCCIncrementalIterator implementation, keys that are outside the time range
still reposition the iterator without time bound. This repositioning no longer
happens, and removing this overhead more than reclaims the RefreshRange
benchmark's regression from the introduction of value blocks that resulted in
packing keys more densely.

This also serves as a step towards cockroachdb#66869.

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                         │    old.txt    │               new.txt               │
                                                         │    sec/op     │   sec/op     vs base                │
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24      150.73µ ± 1%   55.75µ ± 1%  -63.01% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24      16.52µ ± 1%   16.19µ ± 2%   -2.00% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24      13.94µ ± 1%   13.59µ ± 1%   -2.55% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24      13.97µ ± 1%   13.56µ ± 3%   -2.99% (p=0.002 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24      13.96µ ± 1%   13.45µ ± 3%   -3.69% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24    149.95µ ± 1%   55.67µ ± 1%  -62.88% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24     22.05µ ± 1%   21.70µ ± 0%   -1.57% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24     22.08µ ± 1%   21.61µ ± 0%   -2.14% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24     21.91µ ± 1%   21.69µ ± 1%   -0.99% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24    139.27µ ± 1%   55.82µ ± 1%  -59.92% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24     23.76µ ± 1%   23.36µ ± 0%   -1.66% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24     23.85µ ± 1%   23.25µ ± 1%   -2.48% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24    138.94µ ± 1%   55.52µ ± 0%  -60.04% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24     26.68µ ± 1%   26.30µ ± 1%   -1.42% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24    141.39µ ± 1%   55.02µ ± 1%  -61.09% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24      322.35µ ± 0%   51.21µ ± 0%  -84.11% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24      16.35µ ± 1%   16.63µ ± 2%        ~ (p=0.105 n=10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24      17.42µ ± 1%   17.29µ ± 3%        ~ (p=0.796 n=10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24      13.40µ ± 1%   13.45µ ± 1%        ~ (p=0.529 n=10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24      13.59µ ± 2%   13.41µ ± 0%        ~ (p=0.239 n=10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     707.5m ± 1%   111.7m ± 1%  -84.21% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24     21.94µ ± 1%   17.86µ ± 0%  -18.59% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24     13.95µ ± 1%   13.54µ ± 1%   -2.94% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24     13.96µ ± 0%   13.53µ ± 1%   -3.06% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24    395.22m ± 0%   33.81m ± 1%  -91.45% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24     19.42µ ± 1%   19.40µ ± 1%        ~ (p=0.912 n=10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24     19.37µ ± 1%   19.42µ ± 1%        ~ (p=0.247 n=10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24    546.66m ± 3%   48.48m ± 3%  -91.13% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24     21.69µ ± 1%   19.68µ ± 1%   -9.24% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24    546.27m ± 0%   48.73m ± 3%  -91.08% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24        410.1µ ± 0%   221.1µ ± 1%  -46.09% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24       14.84µ ± 1%   14.84µ ± 1%        ~ (p=0.739 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24       14.84µ ± 1%   14.83µ ± 1%        ~ (p=0.615 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24       17.39µ ± 2%   17.51µ ± 2%   +0.68% (p=0.029 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24       16.41µ ± 1%   16.38µ ± 1%        ~ (p=0.971 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24      660.9m ± 0%   140.6m ± 3%  -78.73% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24      20.43µ ± 1%   15.46µ ± 1%  -24.31% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24      18.03µ ± 1%   17.66µ ± 1%   -2.01% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24      16.86µ ± 1%   16.70µ ± 1%   -0.93% (p=0.011 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24      659.5m ± 1%   137.1m ± 3%  -79.21% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24      17.96µ ± 1%   17.65µ ± 1%   -1.77% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24      16.86µ ± 1%   16.66µ ± 1%   -1.23% (p=0.008 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24     900.95µ ± 0%   32.18µ ± 0%  -96.43% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24      14.35µ ± 1%   14.50µ ± 2%   +1.03% (p=0.002 n=10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24    227.235m ± 1%   3.361m ± 1%  -98.52% (p=0.000 n=10)
geomean                                                      136.9µ        73.57µ       -46.24%

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                    │  old2.txt   │              new2.txt               │
                                                    │   sec/op    │   sec/op     vs base                │
MVCCIncrementalIterator/ts=5-24                       24.93m ± 3%   24.53m ± 2%   -1.61% (p=0.004 n=10)
MVCCIncrementalIterator/ts=480-24                     560.4µ ± 1%   453.8µ ± 0%  -19.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=100-24    1.444m ± 0%   1.434m ± 0%   -0.72% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=500-24    1.994m ± 0%   1.974m ± 1%   -1.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=1000-24   2.672m ± 1%   2.649m ± 1%        ~ (p=0.063 n=10)
MVCCIncrementalIteratorForOldData/valueSize=2000-24   4.059m ± 1%   4.009m ± 1%   -1.23% (p=0.011 n=10)
geomean                                               2.754m        2.635m        -4.33%
```

Epic: none
Informs cockroachdb#66869
Close cockroachdb#98881
Release note: none

wip
craig bot pushed a commit that referenced this issue Oct 2, 2023
@jbowens @knz
Merge #111338 #111566 #111579
c133865 
111338: storage: strengthen time-bound iteration guarantees r=sumeerbhola a=jbowens

Previously, the minimum and maximum timestamps used to configure a "time-bound" iterator were only hints. The iterator would surface keys outside those bounds frequently if the keys were contained within a sstable block that also contained keys within bounds. This commit adapts the iterator construction to use Pebble's IterOptions.SkipPoint option to filter out-of-range keys without yielding them to the caller.

This adds a small additional overhead to processing KVs that are ultimately yielded to the user of a MVCCIncrementalIterator, but removes a large overhead to proessing KVs that are ultimately skipped. In the current MVCCIncrementalIterator implementation, keys that are outside the time range still reposition the iterator without time bounds. This repositioning no longer happens, and removing this overhead more than reclaims the RefreshRange benchmark's regression from value blocks packing keys more densely.

This also serves as a step towards #66869.

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                         │    old.txt    │               new.txt               │
                                                         │    sec/op     │   sec/op     vs base                │
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24      150.73µ ± 1%   55.75µ ± 1%  -63.01% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24      16.52µ ± 1%   16.19µ ± 2%   -2.00% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24      13.94µ ± 1%   13.59µ ± 1%   -2.55% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24      13.97µ ± 1%   13.56µ ± 3%   -2.99% (p=0.002 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24      13.96µ ± 1%   13.45µ ± 3%   -3.69% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24    149.95µ ± 1%   55.67µ ± 1%  -62.88% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24     22.05µ ± 1%   21.70µ ± 0%   -1.57% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24     22.08µ ± 1%   21.61µ ± 0%   -2.14% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24     21.91µ ± 1%   21.69µ ± 1%   -0.99% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24    139.27µ ± 1%   55.82µ ± 1%  -59.92% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24     23.76µ ± 1%   23.36µ ± 0%   -1.66% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24     23.85µ ± 1%   23.25µ ± 1%   -2.48% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24    138.94µ ± 1%   55.52µ ± 0%  -60.04% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24     26.68µ ± 1%   26.30µ ± 1%   -1.42% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24    141.39µ ± 1%   55.02µ ± 1%  -61.09% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24      322.35µ ± 0%   51.21µ ± 0%  -84.11% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24      16.35µ ± 1%   16.63µ ± 2%        ~ (p=0.105 n=10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24      17.42µ ± 1%   17.29µ ± 3%        ~ (p=0.796 n=10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24      13.40µ ± 1%   13.45µ ± 1%        ~ (p=0.529 n=10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24      13.59µ ± 2%   13.41µ ± 0%        ~ (p=0.239 n=10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     707.5m ± 1%   111.7m ± 1%  -84.21% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24     21.94µ ± 1%   17.86µ ± 0%  -18.59% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24     13.95µ ± 1%   13.54µ ± 1%   -2.94% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24     13.96µ ± 0%   13.53µ ± 1%   -3.06% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24    395.22m ± 0%   33.81m ± 1%  -91.45% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24     19.42µ ± 1%   19.40µ ± 1%        ~ (p=0.912 n=10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24     19.37µ ± 1%   19.42µ ± 1%        ~ (p=0.247 n=10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24    546.66m ± 3%   48.48m ± 3%  -91.13% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24     21.69µ ± 1%   19.68µ ± 1%   -9.24% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24    546.27m ± 0%   48.73m ± 3%  -91.08% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24        410.1µ ± 0%   221.1µ ± 1%  -46.09% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24       14.84µ ± 1%   14.84µ ± 1%        ~ (p=0.739 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24       14.84µ ± 1%   14.83µ ± 1%        ~ (p=0.615 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24       17.39µ ± 2%   17.51µ ± 2%   +0.68% (p=0.029 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24       16.41µ ± 1%   16.38µ ± 1%        ~ (p=0.971 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24      660.9m ± 0%   140.6m ± 3%  -78.73% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24      20.43µ ± 1%   15.46µ ± 1%  -24.31% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24      18.03µ ± 1%   17.66µ ± 1%   -2.01% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24      16.86µ ± 1%   16.70µ ± 1%   -0.93% (p=0.011 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24      659.5m ± 1%   137.1m ± 3%  -79.21% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24      17.96µ ± 1%   17.65µ ± 1%   -1.77% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24      16.86µ ± 1%   16.66µ ± 1%   -1.23% (p=0.008 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24     900.95µ ± 0%   32.18µ ± 0%  -96.43% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24      14.35µ ± 1%   14.50µ ± 2%   +1.03% (p=0.002 n=10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24    227.235m ± 1%   3.361m ± 1%  -98.52% (p=0.000 n=10)
geomean                                                      136.9µ        73.57µ       -46.24%

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                    │  old2.txt   │              new2.txt               │
                                                    │   sec/op    │   sec/op     vs base                │
MVCCIncrementalIterator/ts=5-24                       24.93m ± 3%   24.53m ± 2%   -1.61% (p=0.004 n=10)
MVCCIncrementalIterator/ts=480-24                     560.4µ ± 1%   453.8µ ± 0%  -19.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=100-24    1.444m ± 0%   1.434m ± 0%   -0.72% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=500-24    1.994m ± 0%   1.974m ± 1%   -1.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=1000-24   2.672m ± 1%   2.649m ± 1%        ~ (p=0.063 n=10)
MVCCIncrementalIteratorForOldData/valueSize=2000-24   4.059m ± 1%   4.009m ± 1%   -1.23% (p=0.011 n=10)
geomean                                               2.754m        2.635m        -4.33%
```

Epic: none
Informs #66869
Close #98881
Release note: none

111566: cli/sql: better report the virtual cluster name in the prompt r=stevendanna a=knz

First commit from #111565.
Fixes #111491.
Epic: CRDB-26691

Prior to this patch, the SQL shell was reporting the name of the
virtual cluster using the `cluster:` option in the connection URL.
This was brittle, as it didn't handle the case where the SQL
connection was routed to a virtual cluster implicitly via the cluster
setting `server.controller.default_target_cluster`.

This commit makes the display more robust by having the server report
the virtual cluster name via the `node_build_info` virtual table,
alongside the cluster ID and organization which were already
communicated in this way.

(No release note because this is new functionality in 23.2)

111579: settings: tweak the comment guidance r=erikgrinaker a=knz

Epic: CRDB-6671

As suggested by `@erikgrinaker` [here](#111378 (comment)) and discussed [here](https://cockroachlabs.slack.com/archives/C05N2J8ACQH/p1696255584272959).

Release note: None

Co-authored-by: Jackson Owens <[email protected]>
Co-authored-by: Raphael 'kena' Poss <[email protected]>
@craig craig bot closed this as completed in f0089ab Oct 2, 2023
THardy98 pushed a commit to THardy98/cockroach that referenced this issue Oct 6, 2023
@jbowens
storage: strengthen time-bound iteration guarantees
3cdcf04 
Previously, the minimum and maximum timestamps used to configure a "time-bound"
iterator were only hints. The iterator would surface keys outside those bounds
frequently if the keys were contained within a sstable block that also
contained keys within bounds. This commit adapts the iterator construction to
use Pebble's IterOptions.SkipPoint option to filter out-of-range keys without
yielding them to the caller.

This adds a small additional overhead to processing KVs that are ultimately
yielded to the user of a MVCCIncrementalIterator, but removes a large overhead
to proessing KVs that are ultimately skipped. In the current
MVCCIncrementalIterator implementation, keys that are outside the time range
still reposition the iterator without time bound. This repositioning no longer
happens, and removing this overhead more than reclaims the RefreshRange
benchmark's regression from the introduction of value blocks that resulted in
packing keys more densely.

This also serves as a step towards cockroachdb#66869.

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                         │    old.txt    │               new.txt               │
                                                         │    sec/op     │   sec/op     vs base                │
RefreshRange/linear-keys/refresh_window=[0.00,0.00]-24      150.73µ ± 1%   55.75µ ± 1%  -63.01% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,50.00]-24      16.52µ ± 1%   16.19µ ± 2%   -2.00% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,75.00]-24      13.94µ ± 1%   13.59µ ± 1%   -2.55% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,95.00]-24      13.97µ ± 1%   13.56µ ± 3%   -2.99% (p=0.002 n=10)
RefreshRange/linear-keys/refresh_window=[0.00,99.00]-24      13.96µ ± 1%   13.45µ ± 3%   -3.69% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,50.00]-24    149.95µ ± 1%   55.67µ ± 1%  -62.88% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,75.00]-24     22.05µ ± 1%   21.70µ ± 0%   -1.57% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,95.00]-24     22.08µ ± 1%   21.61µ ± 0%   -2.14% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[50.00,99.00]-24     21.91µ ± 1%   21.69µ ± 1%   -0.99% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,75.00]-24    139.27µ ± 1%   55.82µ ± 1%  -59.92% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,95.00]-24     23.76µ ± 1%   23.36µ ± 0%   -1.66% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[75.00,99.00]-24     23.85µ ± 1%   23.25µ ± 1%   -2.48% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,95.00]-24    138.94µ ± 1%   55.52µ ± 0%  -60.04% (p=0.000 n=10)
RefreshRange/linear-keys/refresh_window=[95.00,99.00]-24     26.68µ ± 1%   26.30µ ± 1%   -1.42% (p=0.005 n=10)
RefreshRange/linear-keys/refresh_window=[99.00,99.00]-24    141.39µ ± 1%   55.02µ ± 1%  -61.09% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,0.00]-24      322.35µ ± 0%   51.21µ ± 0%  -84.11% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[0.00,50.00]-24      16.35µ ± 1%   16.63µ ± 2%        ~ (p=0.105 n=10)
RefreshRange/random-keys/refresh_window=[0.00,75.00]-24      17.42µ ± 1%   17.29µ ± 3%        ~ (p=0.796 n=10)
RefreshRange/random-keys/refresh_window=[0.00,95.00]-24      13.40µ ± 1%   13.45µ ± 1%        ~ (p=0.529 n=10)
RefreshRange/random-keys/refresh_window=[0.00,99.00]-24      13.59µ ± 2%   13.41µ ± 0%        ~ (p=0.239 n=10)
RefreshRange/random-keys/refresh_window=[50.00,50.00]-24     707.5m ± 1%   111.7m ± 1%  -84.21% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,75.00]-24     21.94µ ± 1%   17.86µ ± 0%  -18.59% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,95.00]-24     13.95µ ± 1%   13.54µ ± 1%   -2.94% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[50.00,99.00]-24     13.96µ ± 0%   13.53µ ± 1%   -3.06% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,75.00]-24    395.22m ± 0%   33.81m ± 1%  -91.45% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[75.00,95.00]-24     19.42µ ± 1%   19.40µ ± 1%        ~ (p=0.912 n=10)
RefreshRange/random-keys/refresh_window=[75.00,99.00]-24     19.37µ ± 1%   19.42µ ± 1%        ~ (p=0.247 n=10)
RefreshRange/random-keys/refresh_window=[95.00,95.00]-24    546.66m ± 3%   48.48m ± 3%  -91.13% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[95.00,99.00]-24     21.69µ ± 1%   19.68µ ± 1%   -9.24% (p=0.000 n=10)
RefreshRange/random-keys/refresh_window=[99.00,99.00]-24    546.27m ± 0%   48.73m ± 3%  -91.08% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,0.00]-24        410.1µ ± 0%   221.1µ ± 1%  -46.09% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,50.00]-24       14.84µ ± 1%   14.84µ ± 1%        ~ (p=0.739 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,75.00]-24       14.84µ ± 1%   14.83µ ± 1%        ~ (p=0.615 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,95.00]-24       17.39µ ± 2%   17.51µ ± 2%   +0.68% (p=0.029 n=10)
RefreshRange/mixed-case/refresh_window=[0.00,99.00]-24       16.41µ ± 1%   16.38µ ± 1%        ~ (p=0.971 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,50.00]-24      660.9m ± 0%   140.6m ± 3%  -78.73% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,75.00]-24      20.43µ ± 1%   15.46µ ± 1%  -24.31% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,95.00]-24      18.03µ ± 1%   17.66µ ± 1%   -2.01% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[50.00,99.00]-24      16.86µ ± 1%   16.70µ ± 1%   -0.93% (p=0.011 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,75.00]-24      659.5m ± 1%   137.1m ± 3%  -79.21% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,95.00]-24      17.96µ ± 1%   17.65µ ± 1%   -1.77% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[75.00,99.00]-24      16.86µ ± 1%   16.66µ ± 1%   -1.23% (p=0.008 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,95.00]-24     900.95µ ± 0%   32.18µ ± 0%  -96.43% (p=0.000 n=10)
RefreshRange/mixed-case/refresh_window=[95.00,99.00]-24      14.35µ ± 1%   14.50µ ± 2%   +1.03% (p=0.002 n=10)
RefreshRange/mixed-case/refresh_window=[99.00,99.00]-24    227.235m ± 1%   3.361m ± 1%  -98.52% (p=0.000 n=10)
geomean                                                      136.9µ        73.57µ       -46.24%

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                    │  old2.txt   │              new2.txt               │
                                                    │   sec/op    │   sec/op     vs base                │
MVCCIncrementalIterator/ts=5-24                       24.93m ± 3%   24.53m ± 2%   -1.61% (p=0.004 n=10)
MVCCIncrementalIterator/ts=480-24                     560.4µ ± 1%   453.8µ ± 0%  -19.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=100-24    1.444m ± 0%   1.434m ± 0%   -0.72% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=500-24    1.994m ± 0%   1.974m ± 1%   -1.02% (p=0.000 n=10)
MVCCIncrementalIteratorForOldData/valueSize=1000-24   2.672m ± 1%   2.649m ± 1%        ~ (p=0.063 n=10)
MVCCIncrementalIteratorForOldData/valueSize=2000-24   4.059m ± 1%   4.009m ± 1%   -1.23% (p=0.011 n=10)
geomean                                               2.754m        2.635m        -4.33%
```

Epic: none
Informs cockroachdb#66869
Close cockroachdb#98881
Release note: none
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbowens jbowens

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-storage Storage Team
Projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage: strengthen time-bound iteration guarantees jbowens/cockroach
6 participants
@erikgrinaker @jbowens @nvanbenschoten @RaduBerinde @andrewbaptist @sumeerbhola