new-test should imply --enable-tests

[dmwit]

I observe the following behavior. Summary: cabal new-test fails to resolve dependencies, while cabal new-test --enable-tests succeeds and runs the tests. I think they should both succeed and run the tests.

% cabal new-test
cabal: Cannot test the package bad-tests-0.1.0.0 because none of the
components are available to build: the test suite 'bad-tests' is not available
because the solver did not find a plan that included the test suites

zsh: exit 1     cabal new-test
% cabal new-test --enable-tests
Resolving dependencies...
Build profile: -w ghc-8.2.1 -O1
In order, the following will be built (use -v for more details):
 - bad-tests-0.1.0.0 (first run)
Preprocessing test suite 'bad-tests' for bad-tests-0.1.0.0..
Building test suite 'bad-tests' for bad-tests-0.1.0.0..
Running 1 test suites...
Test suite bad-tests: RUNNING...
Test suite bad-tests: PASS
Test suite logged to:
<redacted>
1 of 1 test suites (1 of 1 test cases) passed.

The project itself is quite minimal. Here's bad-tests.cabal:

name:                bad-tests
version:             0.1.0.0
build-type:          Simple
cabal-version:       >=1.2

test-suite bad-tests
  type: exitcode-stdio-1.0
  main-is: Main.hs
  build-depends: base, tasty >= 0.10 && < 0.12

Main.hs has just main = return (), and Setup.hs is the standard import Distribution.Simple; main = defaultMain.

[hvr]

Iirc this was a deliberate design choice by @dcoutts to avoid invocations of cabal new-test w/o any flags to suddenly force recomputing a new install-plan (which differs in dependencies), only to have to recompute yet another one when we invoke cabal new-build or cabal new-bench (which btw suffers from the same symptom). The current behaviour causes the install-plans to remain stable when alternating between invocations of build/bench/test.

PS: Also, when we run the test-suite or benchmarks, we want to test the same configuration that we're working on w/ new-build; and not some different arbitrary one.

PS2: The discussion @grayjay linked starting at #3923 (comment) is very relevant

[grayjay]

This issue looks very similar to #3923 and #4421, except that they apply to new-build when it is given a test suite as an argument.

[dmwit]

Given that constraint (which I'm not sure I agree with), I suspect the correct behavior is to default to creating a build plan that includes the tests and benchmarks (even in new-build), then prune those away along with any dependencies that are in the plan exclusively for them if the user hasn't asked for them. And of course one could fall back to the current way of creating an install plan when --disable-tests/--disable-benchmarks is supplied.

A stable build plan is nice. Having a build plan at all in the first place is even nicer.

[phadej]

So we'd need three way flag

• --disable-tests : no tests
• --enable-tests: solve for all tests
• ??? (open for bikesheding, default): enable as much tests as possible?
  EDIT in this case new-test would run what it can solve for.

I really want to have an explicit flag for default behavior (i.e. "neither --disable-tests or --enable-tests is given" isn't good enough)

And the same for benchmarks.

[hvr]

@phadej you also need to take care of the dimension on how much to build by default. Currently this is conflated: cabal new-build --enable-tests solves & enables test components, and builds every enabled target component.

[dmwit]

@phadej Perhaps "local packages" instead of "as much as possible"? Although I still think "all tests" is the right default. It would be just as surprising for cabal new-test mydependency to fail and cabal new-test --enable-tests mydependency to succeed as it is without the explicit testing target.

@hvr We already have the target specification language for specifying what to build, right? And, luckily, the new-* promise is "alpha only, for intrepid users, interface may change", etc. so retrospectively inconsistent behavior like having --enable-tests imply more build targets can be fixed.

[hvr]

@dmwit I'm not worried much about changing the meaning of what all means, but rather that

1. we need to define what cabal new-build all means (which fortunately is not the same as cabal new-build, which rather refers to build the current in-CWD target). What components are considered active? Everything that got solved for?

2. In addition to have a way to say "build everything that has been solved for", we also need a way to say "build everything active except benchmarks and tests" or something like that, as tests & benchmarks are most often not what you want built unless you say cabal new-test or cabal new-bench -- at least I certainly don't want them built everytime I ask "all" to be built (even though I may want to have solved for everything); afaik we have some awkward ways to express that now, but not conveniently.

PS: and also note, that tests/benchmarks can be enabled/disabled at package granularity; it's not a project global setting

[dmwit]

I see. I suppose the problem is roughly this:

1. Coming from old-style cabal, it would be weird if all didn't mean "all the libraries and executables", since that's the default things to build in that style.
2. From a natural-language perspective, it would be weird if all didn't mean "all the things available to be built".
3. We want to avoid claiming too many magic top-level target specifiers, so making more specific magic targets like cabal new-build libraries executables or similar is undesirable.

Does that sound right?

If so, what if all could be given some :-based specifiers for picking which kind of components to build? Then as a first pass, one could imagine something like:

• all: something familiar to old-style users, just libraries and executables
• all:all: really everything
• all:exe, all:lib, generally all:<component-type>: all the artifacts of the given type
• as usual you can give multiple targets, so if you want all the libraries, executables, and tests, you could say all:exe all:lib all:test or all all:test for short.

[hvr]

From a natural-language perspective, it would be weird if all didn't mean "all the things available to be built".

c.f. -Wall vs -Weverything in GHC and Clang ;-)

We want to avoid claiming too many magic top-level target specifiers, so making more specific magic targets like cabal new-build libraries executables or similar is undesirable.

We have this already.... that's what I referred to as awkward/incovenient way; i.e. my most common use-case is to say something like cabal new-build exes libs flibs (not sure whether libs includes sub-libs). The trouble is that it's annoying to type, for what I consider a common use-case. :-)

I also suggest you take a look at the source-code comments in Distribution.Client.TargetSelector to get a better feeling for our rather rich (and partly undocumented -- hint, hint) target selector grammar... :-)

Just to put it out there, what about emulating GHC/Clang's warnings, and have

• cabal new-build all (pragmatic "all")
• cabal new-build everything (true "all")

or alternatively,

• cabal new-build all (pragmatic "all")
• cabal new-build ALL (true "all")

[sjakobi]

Just to point out the absurdity of the current situation:

$ cabal new-build --enable-tests -w ghc-7.10.3
... testsuites are built ...
$ cabal new-test -w ghc-7.10.3
Resolving dependencies...
cabal: Cannot test the package bsb-http-chunked-0.0.0.2 because none of the
components are available to build: the test suite 'tests' and the test suite
'doctests' are not available because the solver did not find a plan that
included the test suites

[hvr]

@sjakobi what's your mental model there? Are you assuming that the --enable-tests flag is stateful? In your example you passed -w ghc-7.10.3 to both new-build and new-test (if you didn't, new-test could easily pick a different GHC version; would that be absurd as well?), and yet you didn't pass --enable-tests to both.

Would you be surprised if

$ cabal new-run -w ghc-7.10.3 test:tests`

failed to find an install-plan because --enable-tests was left off? Wouldn't it be also absurd, if calling cabal new-build; cabal new-test; cabal new-build would cause the last invocation of new-build not to be idempotent?

[phadej]

Short observation: most want to solve for everything (incl tests) but build e.g. only lib part. I mentioned it already before that three state of (`--enable-tests`, `--disable-tests`, and no flag) is confusing. TargetSelector should be really a "SolverConfiguratorTargetSelector", targets we solve for are superset of build targets

…

Sent from my iPhone

On 20 Mar 2018, at 12.32, Herbert Valerio Riedel ***@***.***> wrote: @sjakobi what's your mental model there? Are you assuming that the --enable-tests flag is stateful? In your example you passed -w ghc-7.10.3 to both new-build and new-test (if you didn't, new-test could easily pick a different GHC version; would that be absurd as well?), and yet you didn't pass --enable-tests to both. Would you be surprised if $ cabal new-run -w ghc-7.10.3 test:tests` failed to find an install-plan because --enable-tests was left off? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[fgaz]

Maybe we should just add to the error something like

Maybe you want to use --enable-tests or to set the corresponding option in your cabal.project file

[dmwit]

Adding to the error is certainly the low-effort workaround. But being able to separately specify solver targets and build targets (as suggested by @phadej) -- and then defaulting to a quite wide set of solver targets -- seems like the Right Solution to me.

[hvr]

@dmwit there's just a minor downside to it: I'd have to keep passing --disable-{benchmarks,tests} to the commandline a lot more often, as quite often solving for both tests and benchmarks has no solution for me.

[gelisam]

I really want to have an explicit flag for default behavior (i.e. "neither --disable-tests or --enable-tests is given" isn't good enough)

@phadej, I would like to implement this flag. How about calling it "--only-buildable-tests"?

[simonmar]

I just ran into this: cabal new-test fails with a cryptic error, but cabal new-test --enable-tests works. Whatever the reasoning, this is a counter-intuitive UI. Could it be made to DTRT for common use cases please?

[michaelpj]

We just ran into this on a course. I nearly gave up on new-test, but fortunately it occurred to me to add --enable-tests just in case. I think improving the error message is fine - people probably just want to put tests: true in their cabal.project, but they won't know to do that.

[fgaz]

I think we should at least suggest --enable-tests in the error message before releasing cabal-install 3.0. It may be low effort, but it's quick and would prevent a lot of confusion while the Right Solution is implemented

[erikd]

PR #6058 improves the error message.

@dmwit @simonmar @michaelpj Is this a sufficient improvement?

[sol]

TL;DR: Subtle changes to a package can result in test suites being silently ignored on CI (that's even true for seemingly non-significant things like changing the name of a package).

---

I'm not planning to get too involved in this discussion, but I'm not convinced that this is merely a UI problem. One issue I encountered is that cabal test all can silently ignore test suites.

The implication of this is that subtle changes to a package can result in e.g. CI silently ignoring test suites. This can lead to undetected code breakage down the line.

Given these files:

.
├── cabal.project
├── foo1
│   ├── foo1.cabal
│   └── Main.hs
└── foo2
    └── foo2.cabal

-- cabal.project
packages:
  foo1
  foo2

-- foo1/Main.hs
{-# LANGUAGE CPP #-}
main :: IO ()
main = putStrLn (">>> " <> CURRENT_COMPONENT_ID)

cabal-version:  3.0
-- foo1/foo1.cabal
name:           foo1
version:        0.0.0
build-type:     Simple

executable foo1
  build-depends: base
  main-is: Main.hs

test-suite spec
  type: exitcode-stdio-1.0
  build-depends: base
  main-is: Main.hs

cabal-version:  3.0
-- foo2/foo2.cabal
name:           foo2
version:        0.0.0
build-type:     Simple

test-suite spec
  type: exitcode-stdio-1.0
  build-depends:  base
  main-is: Main.hs
  hs-source-dirs: ../foo1
  -- build-tool-depends: foo1:foo1

cabal test all behaves as I would expect it to:

$ cabal --version 
cabal-install version 3.2.0.0
compiled using version 3.2.0.0 of the Cabal library 
$ cabal test --test-show-details=direct all | grep '>>>'
>>> foo2-0.0.0-inplace-spec
>>> foo1-0.0.0-inplace-spec

However, when you uncomment build-tool-depends in foo2.cabal then foo1:test:spec will be silently ignored:

$ sed -i 's/-- build-tool/build-tool/' foo2/foo2.cabal
$ cabal test --test-show-details=direct all | grep '>>>'
>>> foo2-0.0.0-inplace-spec

The seemingly surprising thing here is: The behavior depends on the package names foo1 and foo2. If you rename foo2 to e.g. bar then cabal test all will continue to run all tests.

I say seemingly because what I suspect is happening here is that the dependency solver "bleeds through" to the UI and influences what the all target encompasses. Assuming that the names of local packages impact the order in which dependencies are solved, it's plausible that renaming a package can result in a change of behavior.

BUT: This took me quite some time to debug and properly understand. So what I'm puzzled is, do we expect everyone who wants to use cabal, even if only occasionally, to go through this "learning experience"?

[sol]

One more thing, I think cabal test all --enable-tests only works as a workaround if you don't use source-repository-packages.

At the point you use source-repository-packages, cabal test all --enable-tests will also run the tests for any source-repository-packages.

At this point I'm puzzled if there is a generic way to run tests so that:

• all tests of a project are run
• tests of dependencies (including source-repository-packages) are not run

I'm pledging USD 10 to the first person who describes a solution that works with cabal only (no find, sed, etc.). Send me your banking details via private message on Twitter if you think your solution qualifies.

[phadej]

At the point you use source-repository-packages, cabal test all --enable-tests will also run the tests for any source-repository-packages.

That is no longer true with cabal-install-3.4.

[sol]

That is no longer true with cabal-install-3.4.

Hey, that's great news! cabal test all --enable-tests seems to reliably run all test with cabal-install-3.4. I still think the UI is unfortunate. But at least we will have something that works. 🎉

@phadej please feel free to claim your bounty.

[phadej]

Donate it to HF (when they open non-corporate donation methods).

[andreasabel]

I just ran into this problem with haskell-hvr/deepseq-generics#12.

Having cabal build --enable-tests succeed but a subsequent cabal test fail is very puzzling.
I resolved it with the suggested tests: True in a cabal.project.local file. (*)

However, I was pessimistic that (*) would make a difference and surprised when it did.

How can you cabal test when tests are disabled? That cabal test would not include --enable-tests---that's what you call a paradox.
[Judging from the (existence of a) long discussion there seem to be internal reasons of some sort why there is a third truth value between tests: False and tests: True, or is there?]

In the case of haskell-hvr/deepseq-generics#12 there is just a single test suite, that apparently builds with cabal build --enable-tests. So I don't understand what cabal test is whining about. Haven't run into this problem before.

[Mikolaj]

cabal test --enable-tests works fine, right? I'd guess the idea was that options should not be implicit. But this is misleading and boilerplate.

[andreasabel]

cabal test fails but increasing verbosity does not provide any insights why:

$ cabal test -v3
File monitor 'config' unchanged.
this build was affected by the following (project) config files:
File monitor 'improved-plan' unchanged.
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:379:14 in Cabal-3.7.0.0-inplace:Distribution.Simple.Utils
Error: cabal: Cannot test the package deepseq-generics-0.2.0.0 because none of
the components are available to build: the test suite 'deepseq-generics-tests'
is not available because the solver did not find a plan that included the test
suites. Force the solver to enable this for all packages by adding the line
'tests: True' to the 'cabal.project.local' file.

cabal test --enable-tests succeeds, yes.

$ cabal test --enable-tests
Resolving dependencies...
Build profile: -w ghc-9.2.1 -O1
In order, the following will be built (use -v for more details):
 - deepseq-generics-0.2.0.0 (lib) (configuration changed)
 - deepseq-generics-0.2.0.0 (test:deepseq-generics-tests) (configuration changed)
Configuring library for deepseq-generics-0.2.0.0..
Preprocessing library for deepseq-generics-0.2.0.0..
Building library for deepseq-generics-0.2.0.0..
Configuring test suite 'deepseq-generics-tests' for deepseq-generics-0.2.0.0..
Preprocessing test suite 'deepseq-generics-tests' for deepseq-generics-0.2.0.0..
Building test suite 'deepseq-generics-tests' for deepseq-generics-0.2.0.0..
Running 1 test suites...
Test suite deepseq-generics-tests: RUNNING...
Test suite deepseq-generics-tests: PASS

[Mikolaj]

Yes, I agree the error messages are confusing, too, even though the hint in the last sentence works. I vote for adding the implicit, default --enable-tests (or something similar) to cabal test and creating --only-buildable-tests option that mimics current default behaviour so that it's not lost. @gelisam: after all those years, are you still up for implementing that? Note that cabal new-configure has since changed semantics, so I don't know if your analysis in #5255 still stands.

[gelisam]

@gelisam: after all those years, are you still up for implementing that?

If I remember correctly, I tried to implement it, but I gave up because the cabal codebase was too complex and confusing for me. Let's give it another shot!

[Mikolaj]

FYI, the difference in cabal v2-configure semantics is that it no longer does a (major) part of what cabal build repeats later on, but it just creates cabal.project.local according to the options given. I don't think cabal test was changed a lot since then, but given that cabal v2-configure is no longer expected before cabal test (nor cabal build), they need to work fine standalone.

Edit: e.g., if you build with cabal build --options-foo you should probably cabal test --options-foo, it's not assumed that you stuck the common options in cabal configure (but you could have in cabal.config.local, manually or via cabal configure).

[Bodigrim]

Why would not we --enable-tests and --enable-benchmarks by default? This would guarantee a stable build plan for cabal build/test/bench and make UI more intuitive. If someone wants to run cabal build and solve only for a library, they can use cabal build --disable-tests.

[fgaz]

It could happen that there are no build plans with tests enabled but there are with tests disabled

[grayjay]

Why would not we --enable-tests and --enable-benchmarks by default? This would guarantee a stable build plan for cabal build/test/bench and make UI more intuitive. If someone wants to run cabal build and solve only for a library, they can use cabal build --disable-tests.

This idea is similar to the solution that the discussion in #7883 was converging on. By default, cabal build would solve for tests but not build them, and cabal test would solve for tests and build them.

[Mikolaj]

It could happen that there are no build plans with tests enabled but there are with tests disabled

Is that a big problem? The user can manually disable tests or benchmarks,

[grayjay]

#7883 is a duplicate of this issue, but it has newer discussion of a possible solution, so I'm going to close this one.