perf: store/cachekv: Correct the naming and implementation for existing benchmarks #10116
Conversation
|
Visit https://dashboard.github.orijtech.com?back=0&pr=10116&remote=false&repo=cosmos%2Fcosmos-sdk to see benchmark details. |
|
Lint issues appear to be from master, not this PR |
There was a problem hiding this comment.
Thank you for this change @ValarDragon! A few things:
- You aren't using b.N; this defeats the purpose of using the benchmarking suite which continually runs until the benchmark is run reliably enough -- this is a must for benchmarks run using -bench= otherwise we'd have to run the code ourselves
- You aren't using keySize -- I presume that this PR is a work-in-progress?
| func BenchmarkCacheKVStoreIterator500(b *testing.B) { benchmarkCacheKVStoreIterator(500, b) } | ||
| func BenchmarkCacheKVStoreIterator1000(b *testing.B) { benchmarkCacheKVStoreIterator(1000, b) } | ||
| func BenchmarkCacheKVStoreIterator10000(b *testing.B) { benchmarkCacheKVStoreIterator(10000, b) } | ||
| func BenchmarkCacheKVStoreIterator50000(b *testing.B) { benchmarkCacheKVStoreIterator(50000, b) } | ||
| func BenchmarkCacheKVStoreIterator100000(b *testing.B) { benchmarkCacheKVStoreIterator(100000, b) } |
There was a problem hiding this comment.
Why aren't we testing with various sizes?
There was a problem hiding this comment.
@ValarDragon can you help get some of these open questions answered. I would like to merge this asap
There was a problem hiding this comment.
This one was (intended to be) answered by the overall reply at the bottom.
The benchmarks atm are structured to do b.N of each operation, with then no further repetitions. At least for Next, this is expected to be a constant time operation / an operation whose size only depends on the parent cache, hence its consistent with the b.N behavior.
This was because the alternative to do things well, would be to make b.N separate, independent stores / independent parents, before starting the timer in order to ensure theres no weird loading with the data cache.
I think we could do size as a separate parameter, but I don't think this is blocking here personally.
| for _, k := range keys { | ||
| kvstore.Set(k, value) | ||
| } |
There was a problem hiding this comment.
We are only running this benchmark once, that's definitely not a reliable way of getting appropriate runs.
There was a problem hiding this comment.
What do you mean? Its doing b.N invocations of set, because thats the number of keys.
Unless the point is that a single set operation is too micro of an operation, and needs to be benchmarked in batches. (E.g. benchmarking the time for 10000 key set operations)
I think that'd make sense iff the goal was to measure growth of Set timing, as number of keys in the store grows.
|
|
||
| iter := kvstore.Iterator(keys[0], keys[b.N]) | ||
|
|
||
| for _ = iter.Key(); iter.Valid(); iter.Next() { |
There was a problem hiding this comment.
Unreliable benchmark; it'll only run say 32 times which defeats the purpose of using b.N .
There was a problem hiding this comment.
What do you mean? Its making b.N calls to next, since its an iterator over b.N keys.
| key := make([]byte, 32) | ||
| value := make([]byte, 32) | ||
| // Use a singleton for value, to not waste time computing it | ||
| value := randSlice(32) |
There was a problem hiding this comment.
We aren't using keysize but we should be.
There was a problem hiding this comment.
Thats for the value though, that doesn't matter there.
What would make more sense imo is hardcoding a value size much greater than expected keysize.
There was a problem hiding this comment.
The value length is only important for cache lines / copying things on stack, so it could be a separate parameter if we want. A default is fine imo. I can change the value to be like 20kb.
I think were pretty far of from being worried about how things grow as value size grows or non-constant sizes. Don't want exponential numbers of benchmark instances, for configurations that aren't that important.
This PR wasn't intended to be profound, but to fix the definitely wrong benchmark naming / code from before, and split out the components to be improved versions of the things it subsumed from before.
|
This is using b.N to setup the exact number of invocations for the hot loop. All the hot loops being measured are of size b.N. Additionally, key size is used for the sizes of the keys here? Its not used for the size of the values, since that doesn't make sense |
Codecov Report
@@ Coverage Diff @@
## master #10116 +/- ##
==========================================
- Coverage 64.33% 64.24% -0.10%
==========================================
Files 592 572 -20
Lines 56008 54230 -1778
==========================================
- Hits 36034 34841 -1193
+ Misses 17912 17408 -504
+ Partials 2062 1981 -81
|
|
@ValarDragon please rebase from the latest on master and apologies for the late replies. |
@odeke-em could leave an approval when ready. Will help saying its ready |
…smos-sdk into dev/correct_cachekv_benchmarks
There was a problem hiding this comment.
Thank you @ValarDragon and well done! Thank you @marbar3778 for the advocacy! Let's ship it, LGTM 🎉🪐
tac0turtle
added
the
A:automerge
…ng benchmarks (cosmos#10116) ## Description Cref discussion in cosmos#10026, updates the CacheKV Benchmark naming and implementation to correspond to whats actually going on, and remove many irrelevant/incorrect components from being in the benchmarks timing. Basically the old Benchmark's iterator creation was very flawed, and never hit the complex case, only repeatedly performing the best case performance of the iterator. Instead it really benchmarked iterator set and next operations. This PR splits out the benchmarks for those two accordingly. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - N/A imo - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [x] confirmed all CI checks have passed - lint failure unrelated ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
* IAVL * perf: store/cachekv: Correct the naming and implementation for existing benchmarks (cosmos#10116) ## Description Cref discussion in cosmos#10026, updates the CacheKV Benchmark naming and implementation to correspond to whats actually going on, and remove many irrelevant/incorrect components from being in the benchmarks timing. Basically the old Benchmark's iterator creation was very flawed, and never hit the complex case, only repeatedly performing the best case performance of the iterator. Instead it really benchmarked iterator set and next operations. This PR splits out the benchmarks for those two accordingly. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - N/A imo - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [x] confirmed all CI checks have passed - lint failure unrelated ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) * store cosmos#10486 * supply cosmos#10393 * supply cosmos#10393 * telemetry cosmos#10334 * module account Co-authored-by: Marko <[email protected]> Co-authored-by: Dev Ojha <[email protected]> Co-authored-by: Emmanuel T Odeke <[email protected]> Co-authored-by: likhita-809 <[email protected]>
…ng benchmarks (cosmos#10116) ## Description Cref discussion in cosmos#10026, updates the CacheKV Benchmark naming and implementation to correspond to whats actually going on, and remove many irrelevant/incorrect components from being in the benchmarks timing. Basically the old Benchmark's iterator creation was very flawed, and never hit the complex case, only repeatedly performing the best case performance of the iterator. Instead it really benchmarked iterator set and next operations. This PR splits out the benchmarks for those two accordingly. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [x] added `!` to the type prefix if API or client breaking change - [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [x] provided a link to the relevant issue or specification - [x] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [x] added a changelog entry to `CHANGELOG.md` - N/A imo - [x] included comments for [documenting Go code](https://blog.golang.org/godoc) - [x] updated the relevant documentation or specification - [x] reviewed "Files changed" and left comments if necessary - [x] confirmed all CI checks have passed - lint failure unrelated ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)
Description
Cref discussion in #10026, updates the CacheKV Benchmark naming and implementation to correspond to whats actually going on, and remove many irrelevant/incorrect components from being in the benchmarks timing.
Basically the old Benchmark's iterator creation was very flawed, and never hit the complex case, only repeatedly performing the best case performance of the iterator. Instead it really benchmarked iterator set and next operations.
This PR splits out the benchmarks for those two accordingly.
Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
!to the type prefix if API or client breaking changeCHANGELOG.md- N/A imoReviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...
!in the type prefix if API or client breaking change