constraint-sets don't encompass tests, benchmarks, or Haddocks

[RyanGlScott]

Currently, if you specify one or more constraint-sets (using haskell-ci's --config) flag, then it will generate a line in .travis.yml for each constraint-set that looks like this:

  # Constraint set no-mtl
- if [ $HCNUMVER -ge 0 ]; then cabal new-build -w ${HC} --disable-tests --disable-benchmarks --constraint='transformers-compat -mtl' all; else echo skipping...; fi

But this only builds the library itself. It doesn't build the tests or benchmarks, and moreover, it doesn't run the tests or build Haddocks, which means that it's possible for things to fail under this constraint-set but not be detected by Travis.

This is currently why I don't use constraint-sets on more projects, as I can't be sure that I don't accidentally introduce some regression in a non-standard flag configuration.

[phadej]

It's a trade-off.

• We (= hackage trustees, maybe?) are very interested in build plans which neither tests or benchmarks affect.
• Adding --enable-tests --enable-benchmarks is more time used per job
• Actually running tests will take even more time

I experienced that for costly jobs (imagine testing lens against all major versions of its dependencies, even one-by-one) Travis time-outs.

And the use-case I had in mind was to test dependency spans, not flag configuration. Therefore current default.

That said, what is done per constraint set can be made configurable, e.g.

constraint-set deepseq-1.4
  ghc: (>= 7.8 && <7.10) || == 8.2.2
  constraints: deepseq ==1.4.*
  run-tests: True

someone have to do it though.

[RyanGlScott]

I would be fine with making this opt-in.

To make my request more concrete, the package which needs this feature is criterion, since its embed-files flag conditionally enables certain code paths with CPP. Without the ability to run the tests under the embed-files constraint set, I can't be confident that it works correctly.

[phadej]

partially fixed by #212; if someone needs to build benchmarks or/and haddocks; please make a PR; it should be straight forward now.

[RyanGlScott]

Without the ability to support all of the main build actions that the main code path supports (including benchmarks and Haddocks), I wouldn't consider this feature to be complete.