cabal v2-install ignores flag --run-tests

[andreasabel]

(I searched whether this issue is already reported, but couldn't find it. If it exists, please tag it with cabal-install: cmd/install.)

It seems like

cabal v2-install --run-tests

simply ignores the option --run-tests. Neither a test-suite is run nor a warning is emitted that this option isn't valid.
Seems like this hasn't been working for 5 years at least, judging from the discussion at https://stackoverflow.com/questions/33040790/how-do-i-run-the-tests-that-are-part-of-an-installed-installing-cabal-package#comment53903271_33041029.

I would find cabal install --run-tests useful because it would run the tests with the sdist-generated package, ensuring that all files needed by the testsuite are packaged up correctly. (One easily forgets adding test files to extra-source-files in the .cabal file.)

I think while this is not working as expected, cabal should issue a warning that the flag is ignored.

I am using a recent version of the development branch:

$ cabal --version
cabal-install version 3.5.0.0
compiled using version 3.5.0.0 of the Cabal library 

[andreasabel]

No triage/comment on this issue?
What is the preferred method then of running the tests in connection with an installation?

[Mikolaj]

@andreasabel: sincere apologies; cabal has a multi-year backlog of issues and PRs that require looking into, so I'm afraid it's mostly self-service for now. Have you discovered anything about this option? Frankly, I forgot it even exists. Perhaps it wasn't ported from v1-install to v2-install? Does it work with v2-build? Actually, why are you using cabal install at all and not cabal build? The StackOverflow discussion seems confused about the transition from v1- to v2-, in particular about using install, where the intention is to build.

[andreasabel]

No need to apologize!

I suppose cabal install --run-tests has been discontinued for a while already.
The reason to try install rather than build is that build works only for local packages. Another reason to try is bug #7577 that cabal test has (and cabal install --run-tests might not have).

Interestingly, in combination with --dependencies-only, we get an error:

$ cabal install --dependencies-only --enable-tests
cabal: --enable-tests was specified, but tests can't be enabled in a remote
package

How about removing --run-tests from the possible options of install and simply give an error (or at least a warning) if this option is supplied?
(I think silently ignoring options is bad for UX as it creates insecurity of why the option didn't have any effect.)

[Mikolaj]

I suppose cabal install --run-tests has been discontinued for a while already.

I guess so. Perhaps the removal of the option was planned together with removal of v1- commands, which is blocked on finding a suitable replacement for v1-install (see https://github.com/haskell/cabal/projects/10).

The reason to try install rather than build is that build works only for local packages.

Oh, so the idea would be to build a package directly from Hackage and run its tests? I'd love to know if it's possible (even by manually executing the test binaries built with cabal install --enable-tests, if any are, in fact, built). If not, that's a desirable feature.

Interestingly, in combination with --dependencies-only, we get an error:

$ cabal install --dependencies-only --enable-tests
cabal: --enable-tests was specified, but tests can't be enabled in a remote
package

That looks like a bug that needs fixing. Option --dependencies-only definitely does not imply working on a package from Hackage (remote).

How about removing --run-tests from the possible options of install and simply give an error (or at least a warning) if this option is supplied?
(I think silently ignoring options is bad for UX as it creates insecurity of why the option didn't have any effect.)

Fully agreed. OTOH, the option will probably just vanish when we remove v1-commands.

[andreasabel]

@Mikolaj wrote:

Fully agreed. OTOH, the option will probably just vanish when we remove v1-commands.

I hope that removal will not happen any time soon. At least not until replacements for v1 workflows have been added.

E.g., v2 does not support (feasibly) -fno-code compilation, which is vital in Agda development to quickly type-check the >400 modules. See #1176.

Is there actually a list/wiki of v1 workflows that still need support by v2? Atm, I reported such on the issue tracker here, but I am worried these get lost in the help of 1400 open issues. I think to make a community-serving decision about the removal of v1, the community should have solutions for their existing setups. To have a structured collection of migration problems and their solutions would help you get hard data about the state of the community.

[Mikolaj]

Noted. I've added the issue to https://github.com/haskell/cabal/projects/12. Please do improve the project. The more coherent it is, the more likely it won't be ignored when postponing the v1- extinction.

[fgaz]

@Mikolaj as mentioned in #1176 (comment) I think the proper (and more general) issue that needs solving to match v1 is #3579. I added it to the project

[Mikolaj]

Oh, great, that's a welcome clue and wider context. Thank you.

[andreasabel]

I suppose cabal install --run-tests has been discontinued for a while already.

To clarify: cabal v1-install --run-tests still runs the tests.

However, if called with an external package, v1-install also silently ignores --run-tests:

$ cabal v1-install goldplate --run-tests
...
Building executable 'goldplate' for goldplate-0.2.0..
...
Linking dist/build/goldplate/goldplate ...
Installing executable goldplate in .../.cabal/bin
Completed    goldplate-0.2.0

I suppose the error given for --dependencies-only would also be appropriate in this situation:

cabal: --enable-tests was specified, but tests can't be enabled in a remote package

[jneira]

I think last comment suggestion is the way to resolve this so PRS welcomed