Formal benchmarks

[rabernat]

At this point we have several different platforms working and a few different use cases. It’s time to start thinking about organizing benchmarks more formally. We need to define

• What we want to measure? Just runtime, or more fine-grained detail?
• What parameters to vary? (n_workers, n_threads, etc.)
• How to automate this? (Shell script, airspeedvelocity, etc.)

Thoughts?

[mrocklin]

I'm interested in this, but I suspect that it is not on the critical path right now. The impression I've gotten is that we should focus more on attracting users than on performance.

[rabernat]

Recruiting more users is definitely important. I would like to improve the docs a bit before we really push on that.

…

Sent from my iPhone

On Nov 30, 2017, at 5:00 PM, Matthew Rocklin ***@***.***> wrote: I'm interested in this, but I suspect that it is not on the critical path right now. The impression I've gotten is that we should focus more on attracting users than on performance. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.