Up for grabs: Port benchmarks to BenchmarkDotNet

[NickCraver]

Currently our performance tests are here: Dapper.Tests/PerformanceTests.cs

These are hard to setup and run. The server side (some tests may use multiple RDBMS platforms) of this with multiple backends is not really avoidable, but re-running them and adding to them should be easier.

I propose we separate these out into another project (Benchmarks, in the root) and port them to use BenchmarkDotNet. This will give us:

• more instrumentation (on GC, etc.)
• more visibility
• better debugging
• more accuracy
• .Net Core support (eventually for all, this depends on providers having a netstandard build)
• the ability to split out those tests cleanly (e.g. 1 comparison test per file)
• easy configuration changes (via [Configuration])
• we can compare multiple versions of EntityFramework, comparing multiple versions in one pass (for once, yay strong naming!)

We can have netstandard and not builds for that project, excluding the non-netstandard providers in #ifdefs for the case where provider or library X doesn't have a .NET Core-compatible build yet.

Related issues:

• Update benchmarks to include EF7 #246 Update benchmarks to include EF7
• Performance of SELECT mapping #274 Performance of SELECT mapping
• (potentially) Dapper doesn't use prepared statements #474 Dapper doesn't use prepared statements
• Update README to show performance comparison of Dapper vs. EF Core #562 Update README to show performance comparison of Dapper vs. EF Core
• Would like to see a Performance Test added for Webmatrix.Data #660 Would like to see a Performance Test added for Webmatrix.Data

Does anyone want to tackle this? It should be pretty self-contained and a good way to explore .NET Core and the benchmarking world. I'll be happy to review. We could use the help.

[mderriey]

I'd love to give it a go!

I'm not familiar at all with the current performance tests setup in Dapper, nor do I have experience with BenchmarkDotNet. I'll try to get up to speed on these.

If you already have a rough idea and/or pointers on how you want it to look like, please let me know.

Also, I won't touch the computer during the Christmas break, so if someone intends to jump on this, please go for it, first arrived first served 😉

Edit: I put together a very early version of a project using BencharkDotNet in the devil-benchmarks branch 😈. Could you have a quick look and let me know if that's roughly what you had in mind?

[philn5d]

Please let me know how I can help. I'm a newb with contributing to OSS, been looking for a way contribute. Likewise won't be in service until after the new year.

[FransBouma]

In https://github.com/FransBouma/RawDataAccessBencher I have many microORMs and full ORMs setup for fetch benchmarks, including Dapper (with easy to use profiling setup too, to profile a given benchmark). I once started to port this to BenchmarkDotNet, but at the time it couldn't deal with a separate config file so I stopped that port. Perhaps an idea to port that instead? It's already a separate project. Not sure if the benchmarks in my project (set fetches, individual fetches, eager load and async) cover all benchmarks you have in this project, but it can be an easy way to get started quickly.

[philn5d]

I've got a minimal start locally of Dapper backed by SQL Express, it's alive!! Would be good to share the work. Perhaps a branch?

[NickCraver]

@phillip-haydon absolutely. Sorry for being quiet here - so, so many libraries in play at the moment needing netstandard love.

[NickCraver]

Done!