cabal test --write-ghc-environment-files=always generates GHC environment files _after_ running tests

[martijnbastiaan]

Describe the bug
When running cabal test with --write-ghc-environment-files=always, a GHC environment file is only written after finishing the tests. Tests relying on the presence of these files (e.g., consumers of the GHC API or tests invoking GHCi) therefore fail to run.

To Reproduce

git clone https://github.com/martijnbastiaan/import-issues-cabal.git
cd import-issues-cabal

This project has a test-suite that:

1. Starts GHCi
2. Executes "import Library"

Which yields:

$ rm -f .ghc.env*
$ cabal test --enable-tests --test-show-details=always --write-ghc-environment-files=always
[..]
GHCi, version 8.10.3: https://www.haskell.org/ghc/  :? for help
Prelude> 
<no location info>: error:
    Could not find module ‘Library’
    It is not a module in the current program, or in any known package.
Prelude> Leaving GHCi.
[..]

$ cabal test --enable-tests --test-show-details=always --write-ghc-environment-files=always
[..]
GHCi, version 8.10.3: https://www.haskell.org/ghc/  :? for help
Loaded package environment from /home/martijn/Code/import-issues-cabal/.ghc.environment.x86_64-linux-8.10.3
Prelude> Prelude Library> Leaving GHCi.
[..]

Interestingly, cabal run does this "correctly":

$ rm -f .ghc.env*
$ cabal v2-run import-issues --write-ghc-environment-files=always
Up to date
GHCi, version 8.10.3: https://www.haskell.org/ghc/  :? for help
Loaded package environment from /home/martijn/Code/import-issues-cabal/.ghc.environment.x86_64-linux-8.10.3
Prelude> Prelude Library> Prelude Library LibraryExtra> Leaving GHCi.

Expected behavior
Like run I expect environment files to be written before a test runs. This is important for consumers of the GHC API of GHCi, such as doctest-parallel or the Clash project.

System information

$ cabal --version
cabal-install version 3.2.0.0
compiled using version 3.2.0.0 of the Cabal library 
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.10.3
$ lsb_release -a
[..]
Description:	Ubuntu 20.04.3 LTS
[..]

[Mikolaj]

This indeed looks like a bug to me. Is there any hint why cabal test behaves so? Any comment in the code that does that?

[martijnbastiaan]

That's a good question. I'll try and dig into the source code tomorrow, but my gut feeling says it's connected to the fact that test can run multiple binaries. Maybe it accumulates all the package ids while running, and therefore can only write the env file in the very end?

[martijnbastiaan]

I think this answers our question: #7890 (comment)

@phadej Do you happen to know why test runs through setup? I can also run tests using cabal run $test, which makes me wonder why cabal test $test wouldn't be basically run + error reporting.

[fgaz]

Do you happen to know why test runs through setup? I can also run tests using cabal run $test, which makes me wonder why cabal test $test wouldn't be basically run + error reporting.

Because there's a Setup.hs test command (and so there are hooks that have to be run) but not a Setup.hs run one:

$ cabal act-as-setup test
Run the 'configure' command first.

$ cabal act-as-setup run
unrecognised command: run (try --help)
Maybe you meant `repl`?

[martijnbastiaan]

Okay, that's too bad. I'm not sure what a good way forward would be, in that case.