Make much more functionality optional.

[thomcc]

Criterion is painful to use in many cases -- bloating the dependency tree so badly that I move benchmarks into another crate excluded from the workspace -- Otherwise just adding a single criterion benchmark expands the dev-dependency tree so much that it's a huge hit to dev ergonomics.

It has many features -- I don't mind these features on their own. I don't think criterion is a bad library, it just has many bad side effects that should be addressed. The use case I'd want to be available is just looking at CLI output, and then if need to dig in, turn on more features and accept the build time, or something.

Anyway I've probably ranted too much to make you want to accept this work. Sorry. Towards the end these patches got pretty sloppy anyway, which is why I stopped... Also I didn't feel like checking if anything other than default-features and no-default-features worked (probably they are slightly broken).

As is it now, it cuts builds on my machine from 27s to 17s for a debug build and 56s to 40s for a release. Next logical step would be removing serde (this code removes a ton of its use), then ideally rayon although it's not quite as bad and might pay for itself in terms of time IDK.

Outputs from full cargo clean; cargo +nightly build -Z timing of before/after here:

https://gist.github.com/thomcc/e1f4bda8bafa4e164483a1b942782a61

[bheisler]

Yes, the dependency tree is quite large. I've tried to keep it as small as I can, but I feel your pain.

The way I intend to deal with this is to create a separate cargo criterion subcommand that invokes the benchmark executables and generates the plots. Then the benchmark executables themselves don't need to link in nearly so much code, and could be limited to just command-line and machine-readable output (probably using serde-json). The benchmarks would be much more lightweight, and the user would only have to compile the plot-generation code when they install the subcommand.

Unfortunately, I don't think I can hide existing code behind a cargo feature (even a default one) without breaking semver compatibility. If I'm going to do that, I might as well break it only once to introduce the subcommand.

[thomcc]

Unfortunately, I don't think I can hide existing code behind a cargo feature (even a default one) without breaking semver compatibility.

I'm pretty sure this is not true. Many crates have made changes like this, including big ones like serde. Consider changes like adding no_std support, and then putting std behind a default-on feature.

[bheisler]

Hey, thanks again for the suggestion. I have some plans in this direction myself in the next breaking-change release - see #426 - but for the 0.3.* series I'll leave things as they are.