kv: insert rows during preparation for BenchmarkKV/Delete #90563
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit resolves a bug and source of confusion in the `BenchmarkKV/Delete` benchmark that affected both the SQL and the Native variant. Before this change, the test was not inserting rows before the Delete benchmark suite, so the delete operations were not deleting anything. The effect of this is that the performance of the benchmark varied wildly depending on whether the SQL optimizer decided to issue blind deletes or non-blind deletes. It also meant that the benchmark unintentionally picked up a large regression caused by 00af7d7. Not inserting rows ahead of the `Delete` benchmark suite was a deliberate choice made by a1e8979. I don't understand the rationale in that commit, which is otherwise unrelated. My best guess is that before that change, the first benchmark iteration deleted real rows and subsequent iterations did not. After the change, at least all iterations were consistently not deleting real rows. With this commit, the performance of the benchmark is consistent across the SQL and Native variants, as well as across SQL statements that hit the auto-commit fast-path and those that do not: ``` name old time/op new time/op delta KV/Scan/Native/rows=10-30 36.0µs ± 4% 35.1µs ± 1% -2.48% (p=0.002 n=9+9) KV/Update/SQL/rows=1000-30 16.9ms ± 1% 16.7ms ± 1% -0.90% (p=0.001 n=10+10) KV/Insert/Native/rows=1-30 122µs ± 3% 123µs ± 2% ~ (p=0.123 n=10+10) KV/Insert/Native/rows=10-30 173µs ± 2% 173µs ± 2% ~ (p=0.684 n=10+10) KV/Insert/Native/rows=100-30 573µs ± 2% 575µs ± 1% ~ (p=0.447 n=10+9) KV/Insert/Native/rows=1000-30 4.03ms ± 2% 4.04ms ± 2% ~ (p=0.796 n=10+10) KV/Insert/Native/rows=10000-30 40.7ms ± 3% 40.5ms ± 3% ~ (p=0.529 n=10+10) KV/Insert/SQL/rows=1-30 426µs ± 1% 426µs ± 1% ~ (p=0.912 n=10+10) KV/Insert/SQL/rows=10-30 533µs ± 2% 533µs ± 1% ~ (p=0.661 n=10+9) KV/Insert/SQL/rows=100-30 1.26ms ± 1% 1.26ms ± 2% ~ (p=0.133 n=10+9) KV/Insert/SQL/rows=1000-30 8.00ms ± 2% 8.01ms ± 2% ~ (p=0.684 n=10+10) KV/Insert/SQL/rows=10000-30 83.1ms ± 4% 83.3ms ± 2% ~ (p=0.684 n=10+10) KV/Update/Native/rows=1-30 189µs ± 2% 188µs ± 1% ~ (p=0.089 n=10+10) KV/Update/Native/rows=10-30 382µs ± 1% 380µs ± 1% ~ (p=0.173 n=10+8) KV/Update/Native/rows=100-30 2.00ms ± 1% 2.00ms ± 1% ~ (p=0.315 n=10+10) KV/Update/Native/rows=1000-30 15.9ms ± 1% 15.9ms ± 1% ~ (p=0.739 n=10+10) KV/Update/Native/rows=10000-30 135ms ± 2% 135ms ± 3% ~ (p=0.796 n=10+10) KV/Update/SQL/rows=1-30 551µs ± 1% 552µs ± 1% ~ (p=0.541 n=9+8) KV/Update/SQL/rows=10-30 842µs ± 2% 843µs ± 1% ~ (p=0.971 n=10+10) KV/Update/SQL/rows=100-30 2.49ms ± 1% 2.49ms ± 1% ~ (p=0.762 n=10+8) KV/Update/SQL/rows=10000-30 195ms ± 3% 195ms ± 4% ~ (p=0.796 n=10+10) KV/Scan/Native/rows=1-30 32.0µs ± 2% 31.8µs ± 4% ~ (p=0.356 n=9+10) KV/Scan/Native/rows=100-30 59.1µs ± 2% 59.3µs ± 5% ~ (p=0.853 n=10+10) KV/Scan/Native/rows=1000-30 273µs ± 1% 272µs ± 2% ~ (p=0.739 n=10+10) KV/Scan/Native/rows=10000-30 2.20ms ± 1% 2.21ms ± 1% ~ (p=0.278 n=9+10) KV/Scan/SQL/rows=1-30 300µs ± 3% 298µs ± 1% ~ (p=0.218 n=10+10) KV/Scan/SQL/rows=10-30 310µs ± 2% 313µs ± 2% ~ (p=0.165 n=10+10) KV/Scan/SQL/rows=100-30 381µs ± 2% 383µs ± 3% ~ (p=0.123 n=10+10) KV/Scan/SQL/rows=1000-30 932µs ± 2% 936µs ± 2% ~ (p=0.113 n=9+10) KV/Scan/SQL/rows=10000-30 5.03ms ± 3% 5.06ms ± 3% ~ (p=0.497 n=10+9) KV/Delete/SQL/rows=1-30 474µs ± 2% 491µs ± 1% +3.52% (p=0.000 n=10+10) KV/Delete/Native/rows=1-30 123µs ± 3% 136µs ± 2% +10.87% (p=0.000 n=10+9) KV/Delete/SQL/rows=10-30 570µs ± 1% 636µs ± 2% +11.63% (p=0.000 n=10+10) KV/Delete/Native/rows=10-30 181µs ± 1% 247µs ± 4% +36.33% (p=0.000 n=9+10) KV/Delete/SQL/rows=100-30 1.22ms ± 2% 2.05ms ± 2% +67.98% (p=0.000 n=10+10) KV/Delete/Native/rows=100-30 634µs ± 1% 1180µs ± 3% +86.12% (p=0.000 n=8+10) KV/Delete/Native/rows=10000-30 48.2ms ± 3% 98.7ms ± 3% +104.84% (p=0.000 n=9+10) KV/Delete/Native/rows=1000-30 4.64ms ± 2% 10.04ms ± 4% +116.36% (p=0.000 n=10+9) KV/Delete/SQL/rows=1000-30 2.71ms ± 0% 23.45ms ± 2% +766.23% (p=0.000 n=9+10) KV/Delete/SQL/rows=10000-30 23.8ms ± 2% 221.9ms ± 9% +833.47% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KV/Update/SQL/rows=100-30 557kB ± 0% 555kB ± 1% -0.32% (p=0.010 n=9+10) KV/Insert/Native/rows=1-30 16.8kB ± 1% 16.8kB ± 1% ~ (p=0.811 n=10+10) KV/Insert/Native/rows=10-30 41.9kB ± 1% 41.9kB ± 1% ~ (p=0.853 n=10+10) KV/Insert/Native/rows=100-30 292kB ± 1% 293kB ± 1% ~ (p=0.631 n=10+10) KV/Insert/Native/rows=1000-30 2.80MB ± 0% 2.81MB ± 1% ~ (p=0.315 n=10+10) KV/Insert/Native/rows=10000-30 34.3MB ± 2% 34.3MB ± 1% ~ (p=0.720 n=9+10) KV/Insert/SQL/rows=1-30 46.9kB ± 1% 46.9kB ± 0% ~ (p=0.905 n=9+10) KV/Insert/SQL/rows=10-30 94.9kB ± 1% 94.6kB ± 0% ~ (p=0.393 n=10+10) KV/Insert/SQL/rows=100-30 534kB ± 1% 534kB ± 0% ~ (p=0.971 n=10+10) KV/Insert/SQL/rows=1000-30 5.26MB ± 1% 5.28MB ± 1% ~ (p=0.280 n=10+10) KV/Insert/SQL/rows=10000-30 66.1MB ± 1% 66.2MB ± 2% ~ (p=0.912 n=10+10) KV/Update/Native/rows=1-30 23.4kB ± 1% 23.5kB ± 1% ~ (p=0.796 n=10+10) KV/Update/Native/rows=10-30 70.5kB ± 0% 70.6kB ± 1% ~ (p=0.052 n=10+10) KV/Update/Native/rows=100-30 519kB ± 0% 519kB ± 0% ~ (p=0.529 n=10+10) KV/Update/Native/rows=1000-30 5.12MB ± 1% 5.11MB ± 0% ~ (p=0.661 n=10+9) KV/Update/Native/rows=10000-30 64.3MB ± 1% 64.0MB ± 2% ~ (p=0.143 n=10+10) KV/Update/SQL/rows=1-30 53.9kB ± 0% 53.8kB ± 1% ~ (p=0.133 n=10+9) KV/Update/SQL/rows=10-30 121kB ± 1% 121kB ± 1% ~ (p=0.853 n=10+10) KV/Update/SQL/rows=1000-30 4.98MB ± 1% 4.96MB ± 1% ~ (p=0.247 n=10+10) KV/Update/SQL/rows=10000-30 143MB ± 1% 143MB ± 1% ~ (p=0.315 n=10+10) KV/Delete/SQL/rows=1-30 54.4kB ± 1% 54.4kB ± 1% ~ (p=0.736 n=9+10) KV/Scan/Native/rows=1-30 7.72kB ± 0% 7.71kB ± 0% ~ (p=0.068 n=10+9) KV/Scan/Native/rows=10-30 9.15kB ± 0% 9.15kB ± 1% ~ (p=1.000 n=10+9) KV/Scan/Native/rows=100-30 22.0kB ± 0% 22.0kB ± 0% ~ (p=0.305 n=10+9) KV/Scan/Native/rows=1000-30 175kB ± 0% 175kB ± 0% ~ (p=0.549 n=9+10) KV/Scan/Native/rows=10000-30 1.53MB ± 0% 1.54MB ± 0% ~ (p=0.481 n=10+10) KV/Scan/SQL/rows=1-30 26.8kB ± 1% 26.8kB ± 1% ~ (p=1.000 n=9+9) KV/Scan/SQL/rows=10-30 28.2kB ± 1% 28.2kB ± 0% ~ (p=0.971 n=10+10) KV/Scan/SQL/rows=100-30 40.3kB ± 1% 40.3kB ± 1% ~ (p=0.905 n=10+9) KV/Scan/SQL/rows=1000-30 254kB ± 0% 254kB ± 0% ~ (p=0.853 n=10+10) KV/Scan/SQL/rows=10000-30 1.12MB ± 0% 1.12MB ± 0% ~ (p=0.218 n=10+10) KV/Delete/Native/rows=1-30 16.4kB ± 1% 16.6kB ± 1% +1.18% (p=0.005 n=10+10) KV/Delete/SQL/rows=10-30 87.8kB ± 1% 89.0kB ± 1% +1.38% (p=0.000 n=10+10) KV/Delete/Native/rows=1000-30 2.45MB ± 1% 2.57MB ± 1% +5.04% (p=0.000 n=10+9) KV/Delete/Native/rows=10000-30 30.5MB ± 2% 32.0MB ± 2% +5.11% (p=0.000 n=9+10) KV/Delete/Native/rows=10-30 38.6kB ± 1% 41.0kB ± 4% +6.34% (p=0.000 n=10+10) KV/Delete/Native/rows=100-30 256kB ± 0% 276kB ± 1% +7.65% (p=0.000 n=10+10) KV/Delete/SQL/rows=100-30 415kB ± 1% 506kB ± 1% +21.77% (p=0.000 n=9+10) KV/Delete/SQL/rows=1000-30 1.15MB ± 0% 11.12MB ± 1% +865.80% (p=0.000 n=10+8) KV/Delete/SQL/rows=10000-30 13.3MB ± 0% 141.6MB ± 2% +961.61% (p=0.000 n=10+10) name old allocs/op new allocs/op delta KV/Insert/Native/rows=100-30 1.65k ± 1% 1.64k ± 0% -0.33% (p=0.024 n=10+9) KV/Insert/Native/rows=1-30 134 ± 1% 134 ± 1% ~ (p=0.637 n=10+10) KV/Insert/Native/rows=10-30 289 ± 1% 290 ± 1% ~ (p=0.672 n=9+10) KV/Insert/Native/rows=1000-30 14.9k ± 0% 14.9k ± 0% ~ (p=0.127 n=10+10) KV/Insert/Native/rows=10000-30 148k ± 1% 147k ± 1% ~ (p=0.143 n=10+10) KV/Insert/SQL/rows=1-30 401 ± 0% 401 ± 0% ~ (p=0.335 n=9+10) KV/Insert/SQL/rows=10-30 638 ± 2% 633 ± 0% ~ (p=0.133 n=10+10) KV/Insert/SQL/rows=100-30 2.64k ± 0% 2.64k ± 0% ~ (p=0.321 n=9+10) KV/Insert/SQL/rows=1000-30 23.5k ± 0% 23.6k ± 0% ~ (p=0.138 n=10+10) KV/Insert/SQL/rows=10000-30 237k ± 1% 237k ± 1% ~ (p=0.905 n=10+9) KV/Update/Native/rows=1-30 187 ± 0% 187 ± 1% ~ (p=0.912 n=6+10) KV/Update/Native/rows=10-30 470 ± 1% 470 ± 0% ~ (p=0.406 n=10+10) KV/Update/Native/rows=100-30 2.87k ± 0% 2.87k ± 0% ~ (p=0.810 n=10+10) KV/Update/Native/rows=1000-30 27.7k ± 0% 27.7k ± 0% ~ (p=0.328 n=9+9) KV/Update/Native/rows=10000-30 285k ± 0% 285k ± 0% ~ (p=0.927 n=10+10) KV/Update/SQL/rows=1-30 548 ± 0% 548 ± 1% ~ (p=0.709 n=10+9) KV/Update/SQL/rows=10-30 914 ± 0% 916 ± 1% ~ (p=0.082 n=10+10) KV/Update/SQL/rows=100-30 6.42k ± 0% 6.41k ± 0% ~ (p=0.090 n=9+10) KV/Update/SQL/rows=1000-30 53.8k ± 0% 53.8k ± 0% ~ (p=0.927 n=10+10) KV/Update/SQL/rows=10000-30 700k ± 1% 697k ± 2% ~ (p=0.481 n=10+10) KV/Scan/Native/rows=1-30 53.0 ± 0% 53.0 ± 0% ~ (all equal) KV/Scan/Native/rows=10-30 58.0 ± 0% 58.0 ± 0% ~ (all equal) KV/Scan/Native/rows=100-30 65.6 ± 1% 66.0 ± 0% ~ (p=0.108 n=10+9) KV/Scan/Native/rows=1000-30 109 ± 4% 110 ± 6% ~ (p=0.561 n=9+10) KV/Scan/Native/rows=10000-30 453 ± 5% 454 ± 2% ~ (p=0.644 n=10+10) KV/Scan/SQL/rows=1-30 292 ± 1% 292 ± 0% ~ (p=0.875 n=9+7) KV/Scan/SQL/rows=10-30 320 ± 1% 320 ± 0% ~ (p=1.000 n=10+10) KV/Scan/SQL/rows=100-30 611 ± 1% 612 ± 0% ~ (p=0.221 n=10+9) KV/Scan/SQL/rows=1000-30 4.18k ± 0% 4.18k ± 0% ~ (p=0.305 n=10+10) KV/Scan/SQL/rows=10000-30 40.9k ± 0% 40.9k ± 0% ~ (p=0.726 n=10+10) KV/Delete/SQL/rows=1-30 438 ± 0% 443 ± 1% +1.06% (p=0.000 n=9+10) KV/Delete/Native/rows=1-30 133 ± 1% 136 ± 1% +2.19% (p=0.000 n=10+10) KV/Delete/SQL/rows=10-30 657 ± 2% 679 ± 0% +3.40% (p=0.000 n=10+9) KV/Delete/Native/rows=10-30 259 ± 1% 286 ± 1% +10.81% (p=0.000 n=10+9) KV/Delete/Native/rows=100-30 1.35k ± 0% 1.62k ± 1% +20.63% (p=0.000 n=9+10) KV/Delete/Native/rows=10000-30 119k ± 1% 147k ± 0% +23.18% (p=0.000 n=10+8) KV/Delete/Native/rows=1000-30 12.0k ± 0% 14.9k ± 2% +24.57% (p=0.000 n=10+10) KV/Delete/SQL/rows=100-30 2.45k ± 0% 6.16k ± 0% +151.81% (p=0.000 n=9+10) KV/Delete/SQL/rows=1000-30 7.95k ± 0% 70.92k ± 0% +792.16% (p=0.000 n=10+9) KV/Delete/SQL/rows=10000-30 74.9k ± 0% 773.9k ±12% +933.72% (p=0.000 n=8+10) ``` Epic: None. Release note: None.
|
This change is |
|
bors r+ |
|
Build succeeded: |
18 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit resolves a bug and source of confusion in the
BenchmarkKV/Deletebenchmark that affected both the SQL and the Native variant. Before this change, the test was not inserting rows before the Delete benchmark suite, so the delete operations were not deleting anything. The effect of this is that the performance of the benchmark varied wildly depending on whether the SQL optimizer decided to issue blind deletes or non-blind deletes. It also meant that the benchmark unintentionally picked up a large regression caused by 00af7d7.Not inserting rows ahead of the
Deletebenchmark suite was a deliberate choice made by a1e8979. I don't understand the rationale in that commit, which is otherwise unrelated. My best guess is that before that change, the first benchmark iteration deleted real rows and subsequent iterations did not. After the change, at least all iterations were consistently not deleting real rows.With this commit, the performance of the benchmark is consistent across the SQL and Native variants, as well as across SQL statements that hit the auto-commit fast-path and those that do not:
Epic: None.
Release note: None.