Test coverage for package whose test suite is in another package.

[NorfairKing]

General summary/comments (optional)

Coverage reports are broken when you use a test suite in one package to test another that doesn't have a test suite.

Steps to reproduce

1. Make package foobar with code in a library foobar:lib:foobar.
2. Make package foobar-gen with generators and testutils in a library foobar-gen:lib:foobar-gen
3. Make test suite foobar-gen:test:foobar-test to test foobar using the generators in foobar-gen.

Run stack test foobar foobar-gen --coverage. Find that there is no coverage report for any code in foobar.

Expected

Coverage report for both foobar and foobar-gen

Actual

No coverage report for foobar.

Stack version

$ stack --version
2.7.3 x86_64 hpack-0.34.6

Method of installation

• Nixpkgs

[NorfairKing]

Workaround: make a dummy test suite foobar:test:foobar-dummy-test with main = pure () that depends on foobar:lib:foobar.

[NorfairKing]

@qrilka Any opinions?

[NorfairKing]

@mpilgrem any opinions?

[mpilgrem]

I don't have much experience at all with automated testing, so I suspect I cannot help. My starting point is https://docs.haskellstack.org/en/stable/build_command/#target-syntax for targets and https://docs.haskellstack.org/en/stable/coverage/#code-coverage for the --coverage option.

The targets documentation explains that stack build takes 'zero or more' targets. The --test flag (equivalently, stack test) explains that it builds the library, executables, and test executables components and then runs the test executables.

The coverage documentation explains that it is 'quite streamlined', and its scope (only the libraries that the test executables link against).

So, I wonder if stack test foobar foobar-gen --coverage (with two targets) is the equivalent of:

stack test foobar --coverage
stack test foobar-gen --coverage

and that might explain what you experience?

[NorfairKing]

and that might explain what you experience?

Yes that explains, but it's no less wrong not what I want. :(
There's no way to get the coverage report for foobar in that case.

[mpilgrem]

I've added an 'enhancement' label, then. Coming to this cold, it seems to me that --coverage currently reports the extent to which each package tests its own library. A tool that reports the extent to which each package also tests the library of one or more different packages (but not others) seems to me to be quite a different 'coverage' model.

[NorfairKing]

I just realised it's not as simple as your comment above suggests.
When I run stack test foobar foobar-gen --coverage with my workaround, I also get a coverage report for foobar with coverage from the tests of foobar-gen.

[mpilgrem]

My little knowledge may be a dangerous thing. I've started again, from the beginning. As you will see below, I don't seem to be experiencing the problem you are experiencing.

Setup

I am testing this problem with a simple 3-package project named multi, with folder and file structure:

\multi
  \packageA
    package.yaml
    packageA.cabal
  \test
    Spec.hs
  <other source code and other files for packageA>
  \packageB
    package.yaml
    packageB.cabal
  \test
    Spec1.hs
    Spec2.hs
  <other source code and other files for packageB>
  \packageC
    package.yaml
    packageC.cabal
  \test
    Spec.hs
  <other source code and other files for packageC>
  stack.yaml

The stack.yaml is as follows (I am using GHC 9.0.2):

resolver: lts-19.5
packages:
- packageA
- packageB
- packageC

The source code for the individual packages is based on the simple example template created by stack new, but I've re-named the library module Lib, LibA, LibB or LibC and changed the code to export two functions. For example, for package A:

module LibA
    ( someFunc1A
    , someFunc2A
    ) where

someFunc1A :: IO ()
someFunc1A = putStrLn "someFunc1A"

someFunc2A :: IO ()
someFunc2A = putStrLn "someFunc2A"

I've changed the code for the tests for package A to (that is, no actions):

main :: IO ()
main = pure ()

I've changed the code for the tests for package B to have two test executables, each of which tests one of someFunc1B or someFunc2B. For example, Spec1.hs is:

import LibB

main :: IO ()
main = do
  putStrLn "Testing package B library - someFunc1B"
  someFunc1B

and the tests: key of B's package.yaml is:

tests:
  packageB-test1:
    main:                Spec1.hs
    source-dirs:         test
    when:
    - condition: false
      other-modules: Spec2
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - packageB
  packageB-test2:
    main:                Spec2.hs
    source-dirs:         test
    when:
    - condition: false
      other-modules: Spec1
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - packageB

I've changed the code for the tests for package C to test parts of C and A:

import LibA 
import LibC

main :: IO ()
main = do
  putStrLn "Testing package C library - someFunc1C"
  someFunc1C
  putStrLn "Testing package A library - someFunc2A"
  someFunc2A

The tests: key of C's package.yaml reflects that the tests depend on A and C:

tests:
  packageC-test:
    main:                Spec.hs
    source-dirs:         test
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - packageC
    - packageA

Results

I think that setup is equivalent to your workaround. I stack purge to ensure that I am starting with a clean slate. If I stack test --coverage that (from the \multi root folder), I get (a) a coverage report for each test executable in A, B and C (the report for A being text-only and warning that the test did not consider any code) and (b) a unified coverage report that picks up that (1) the B tests tested all of B and (2) the C test tested part of A as well as part of C. In the folders of \multi\.stack-work\install\ \hpc there is a *.tix file for each test executable (including packageA-test.tix). The packageA-test.tix file contains only Tix [].

I then comment out the tests: key in A's package.yaml. This results in a packageA.cabal that also lacks a tests stanza. I think that corresponds to your initial problem. I stack purge again, to ensure a clean slate. If I stack-test --coverage that (again, from the \multi root), I get:

• a coverage report for each test executable in B and C (as expected, there is no coverage report for A)
• the same unified coverage report as before. That is, one that picks up that C tested part of A as well as part of C. I don't seem to be experiencing your problem.

In the folders of \multi\.stack-work\install\ \hpc there is a *.tix file for each test executable of B and C (as expected, there is none for A).

If I change the code for the tests for package C to test all of A, to:

import LibA 
import LibC

main :: IO ()
main = do
  putStrLn "Testing package C library - someFunc1C"
  someFunc1C
  putStrLn "Testing package A library - someFunc1A"
  someFunc1A
  putStrLn "Testing package A library - someFunc2A"
  someFunc2A

again, with no testing in package A, the unified coverage report picks up that C has tested all of A.

Perhaps this is explained by different stack versions. You are using 2.7.3. I am using one built from the current source code in branch master:

Version 2.7.6, Git revision dcf7bb31815c7505e809a1707909832e02614501 (8439 commits) RELEASE-CANDIDATE x86_64
Compiled with:
- Cabal-3.2.1.0
- Win32-2.6.1.0
- base-4.14.1.0

[NorfairKing]

I've dug into this example of yours, great job by the way!
I could not reproduce the problem with stack test --coverage (which made me feel a bit guilty because you did such a nice job trying to reproduce this in your example repo), but then I could with stack test packageA packageB --coverage.
I think we're getting close to figuring this out...

[mpilgrem]

I think you likely meant stack test packageA packageC --coverage. Using the example in my repo mpilgrem/stack-coverage-tests, with no test executable for A, I get (extracts only):

❯ stack test packageA packageC --coverage
packageA> build (lib + exe)
packageC> build (lib + exe + test)
packageC> test (suite: packageC-test)

packageC> Testing package C library - someFunc1C
packageC> someFunc1C
packageC> Testing package A library - someFunc2A
packageC> someFunc2A
packageC> Test suite packageC-test passed
Generating coverage report for packageC's test-suite "packageC-test"
 50% expressions used (2/4)
100% boolean coverage (0/0)
     100% guards (0/0)
     100% 'if' conditions (0/0)
     100% qualifiers (0/0)
100% alternatives used (0/0)
100% local declarations used (0/0)
 50% top-level declarations used (1/2)
The coverage report for packageC's test-suite "packageC-test" is available at D:\Users\mike\Code\GitHub\stack-coverage-tests\multi\.stack-work\install\c30669b5\hpc\packageC\packageC-test\hpc_index.html
Completed 3 action(s).
Only one tix file found in D:\Users\mike\Code\GitHub\stack-coverage-tests\multi\.stack-work\install\c30669b5\hpc\, so not generating a unified coverage report.

An index of the generated HTML coverage reports is available at D:\Users\mike\Code\GitHub\stack-coverage-tests\multi\.stack-work\install\c30669b5\hpc\index.html

It seems that with only one *.tix file generated (for C), there is no attempt at a unified coverage report, so C's testing of A is 'lost'. @NorfairKing, is that the problem you were experiencing?

[NorfairKing]

Jup! Looks like that's it :D

[mpilgrem]

OK. Some relevant code is in Stack.Coverage.generateHpcUnifiedReport:

if length tixFiles < 2
        then logInfo $
            (if null tixFiles then "No tix files" else "Only one tix file") <>
            " found in " <>
            fromString (toFilePath outputDir) <>
            ", so not generating a unified coverage report."
        else do
            let report = "unified report"
            mreportPath <- generateUnionReport report reportDir tixFiles
            forM_ mreportPath (displayReportPath report . pretty)

If (for the moment) I disable that by replacing the test with if length tixFiles < 1, then your problem is solved. So, it seems there are two possible solutions:

1. (simple) generate a unified coverage report, even in circumstances where it may be redundant; or
2. (more complex) have a more complex test to work out if a unified coverage report would genuinely be redundant.

I will propose a pull request for the simple solution, as an interim solution.