stack test --coverage "did not consider any code", fails to produce coverage report #3997

DanBurton · 2018-04-29T22:36:28Z

Steps to reproduce

stack unpack lens-family-th-0.5.0.1 && cd lens-family-th-0.5.0.1
echo 'resolver: nightly-2018-04-28' > stack.yaml
stack test --coverage

Expected

I expect an accurate coverage report to be generated.

Actual

Error: The coverage report for lens-family-th's test-suite "lens-family-th-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.

Note that:

The .cabal file library target has hs-source-dirs: src, while the test-suite target has hs-source-dirs: test. The test suite build-depends on the lib. (thus avoiding Warn when a test-suite / executable / benchmark also builds the library code? #1008)
This command is run in a fresh stack unpack, with no previous .stack-work state in the directory. (thus avoiding Correct "stack test --coverage" results requires a "stack clean" #1411. Even so, I have tried the same command with --ghc-options -fforce-recomp; same result.)

Here's the verbose output:

$ stack test --coverage --verbose
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2
2018-04-29 15:29:03.335683: [debug] Checking for project config at: /Users/dan/scratch/lens-family-th-0.5.0.1/stack.yaml
@(src/Stack/Config.hs:850:9)
2018-04-29 15:29:03.336823: [debug] Loading project config file stack.yaml
@(src/Stack/Config.hs:876:13)
2018-04-29 15:29:03.338509: [debug] Decoding build plan from: /Users/dan/.stack/build-plan/nightly-2018-04-28.yaml
@(src/Stack/Snapshot.hs:164:5)
2018-04-29 15:29:03.338610: [debug] Trying to decode /Users/dan/.stack/build-plan-cache/nightly-2018-04-28.cache
@(src/Stack/Snapshot.hs:156:32)
2018-04-29 15:29:03.340556: [debug] Success decoding /Users/dan/.stack/build-plan-cache/nightly-2018-04-28.cache
@(src/Stack/Snapshot.hs:156:32)
2018-04-29 15:29:03.344153: [debug] Potential GHC builds: standard
@(src/Stack/Setup.hs:632:9)
2018-04-29 15:29:03.344251: [debug] Found already installed GHC builds: standard
@(src/Stack/Setup.hs:458:13)
2018-04-29 15:29:03.345312: [debug] Getting global package database location
@(src/Stack/GhcPkg.hs:47:5)
2018-04-29 15:29:03.345534: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db list --global
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.346062: [debug] Asking GHC for its version
@(src/Stack/Setup/Installed.hs:98:13)
2018-04-29 15:29:03.346174: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc --numeric-version
@(src/Stack/Setup/Installed.hs:99:19)
2018-04-29 15:29:03.346058: [debug] Getting Cabal package version
@(src/Stack/GhcPkg.hs:180:5)
2018-04-29 15:29:03.346651: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.408118: [debug] Process finished in 62ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db list --global
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.408355: [debug] Process finished in 62ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.417059: [debug] Process finished in 71ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc --numeric-version
@(src/Stack/Setup/Installed.hs:99:19)
2018-04-29 15:29:03.417319: [debug] GHC version is: ghc-8.4.2
@(src/Stack/Setup/Installed.hs:102:13)
2018-04-29 15:29:03.417399: [debug] Resolving package entries
@(src/Stack/Setup.hs:256:5)
2018-04-29 15:29:03.417497: [debug] Trying to decode /Users/dan/.stack/loaded-snapshot-cache/x86_64-osx/ghc-8.4.2/nightly-2018-04-28.cache
@(src/Stack/Snapshot.hs:384:9)
2018-04-29 15:29:03.442379: [debug] Success decoding /Users/dan/.stack/loaded-snapshot-cache/x86_64-osx/ghc-8.4.2/nightly-2018-04-28.cache
@(src/Stack/Snapshot.hs:384:9)
2018-04-29 15:29:03.443385: [debug] Starting to execute command inside EnvConfig
@(src/Stack/Runners.hs:171:18)
2018-04-29 15:29:03.443443: [debug] Parsing the targets
@(src/Stack/Build/Target.hs:460:3)
2018-04-29 15:29:03.453604: [debug] Start: getPackageFiles /Users/dan/scratch/lens-family-th-0.5.0.1/lens-family-th.cabal
@(src/Stack/Package.hs:332:21)
2018-04-29 15:29:03.456117: [debug] Finished in 2ms: getPackageFiles /Users/dan/scratch/lens-family-th-0.5.0.1/lens-family-th.cabal
@(src/Stack/Package.hs:332:21)
2018-04-29 15:29:03.456768: [debug] Finding out which packages are already installed
@(src/Stack/Build/Installed.hs:57:5)
2018-04-29 15:29:03.456925: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.518075: [debug] Process finished in 61ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.518691: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --user --no-user-package-db --package-db /Users/dan/.stack/snapshots/x86_64-osx/nightly-2018-04-28/8.4.2/pkgdb dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.642160: [debug] Process finished in 123ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --user --no-user-package-db --package-db /Users/dan/.stack/snapshots/x86_64-osx/nightly-2018-04-28/8.4.2/pkgdb dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.644184: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --user --no-user-package-db --package-db /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/pkgdb dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.695703: [debug] Process finished in 51ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --user --no-user-package-db --package-db /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/pkgdb dump --expand-pkgroot
@(src/Stack/Prelude.hs:117:3)
2018-04-29 15:29:03.696136: [debug] Constructing the build plan
@(src/Stack/Build/ConstructPlan.hs:186:5)
2018-04-29 15:29:03.699275: [debug] Checking if we are going to build multiple executables with the same name
@(src/Stack/Build.hs:177:5)
2018-04-29 15:29:03.699354: [debug] Executing the build plan
@(src/Stack/Build/Execute.hs:491:5)
2018-04-29 15:29:03.699977: [debug] Getting global package database location
@(src/Stack/GhcPkg.hs:47:5)
2018-04-29 15:29:03.700035: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db list --global
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.752898: [debug] Process finished in 53ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/ghc-pkg --no-user-package-db list --global
@(src/Stack/GhcPkg.hs:76:10)
2018-04-29 15:29:03.755047: [info] lens-family-th-0.5.0.1: test (suite: lens-family-th-test)
@(src/Stack/Build/Execute.hs:873:23)
2018-04-29 15:29:03.755142: [info] 
@(src/Stack/Build/Execute.hs:1701:29)
2018-04-29 15:29:03.755364: [debug] Run process within /Users/dan/scratch/lens-family-th-0.5.0.1/: /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/build/lens-family-th-test/lens-family-th-test
@(src/Stack/Build/Execute.hs:1711:27)

makeLenses
  makes lenses that function with lens-family operators
makeTraversals
  makes traversals that function with lens-family operators

Finished in 0.0002 seconds
2 examples, 0 failures
2018-04-29 15:29:03.774779: [debug] Process finished in 19ms: /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/build/lens-family-th-test/lens-family-th-test
@(src/Stack/Build/Execute.hs:1711:27)
2018-04-29 15:29:03.774891: [info] 
@(src/Stack/Build/Execute.hs:1724:52)
2018-04-29 15:29:03.779828: [info] lens-family-th-0.5.0.1: Test suite lens-family-th-test passed
@(src/Stack/Build/Execute.hs:873:23)
2018-04-29 15:29:03.779972: [debug] Scanning /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/package.conf.inplace/ for files matching lens-family-th-0.5.0.1
@(src/Stack/Coverage.hs:449:13)
2018-04-29 15:29:03.780269: [debug] ["/Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/package.conf.inplace/package.cache","/Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/package.conf.inplace/package.cache.lock","/Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/package.conf.inplace/lens-family-th-0.5.0.1-9zItzPrK3BVHwIqPNZ0M0m.conf"]
@(src/Stack/Coverage.hs:451:13)
2018-04-29 15:29:03.780644: [info] Generating coverage report for lens-family-th's test-suite "lens-family-th-test"
@(src/Stack/Coverage.hs:170:13)
2018-04-29 15:29:03.780808: [debug] Run process: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/hpc report /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/hpc/lens-family-th/lens-family-th-test/lens-family-th-test.tix --srcdir /Users/dan/scratch/lens-family-th-0.5.0.1 --hpcdir .stack-work/dist/x86_64-osx/Cabal-2.2.0.1/hpc --reset-hpcdirs --include lens-family-th-0.5.0.1-9zItzPrK3BVHwIqPNZ0M0m:
@(src/Stack/Coverage.hs:172:17)
2018-04-29 15:29:03.816748: [debug] Process finished in 36ms: /Users/dan/.stack/programs/x86_64-osx/ghc-8.4.2/bin/hpc report /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/hpc/lens-family-th/lens-family-th-test/lens-family-th-test.tix --srcdir /Users/dan/scratch/lens-family-th-0.5.0.1 --hpcdir .stack-work/dist/x86_64-osx/Cabal-2.2.0.1/hpc --reset-hpcdirs --include lens-family-th-0.5.0.1-9zItzPrK3BVHwIqPNZ0M0m:
@(src/Stack/Coverage.hs:172:17)
2018-04-29 15:29:03.816911: [error] Error: The coverage report for lens-family-th's test-suite "lens-family-th-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.
@(src/Stack/Coverage.hs:191:21)
2018-04-29 15:29:03.817576: [debug] Encoding /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/stack-test-success
@(src/Stack/Build/Cache.hs:220:7)
2018-04-29 15:29:03.818120: [debug] Finished writing /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/stack-test-success
@(src/Stack/Build/Cache.hs:220:7)
2018-04-29 15:29:03.819043: [info] Only one tix file found in /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/hpc/, so not generating a unified coverage report.
@(src/Stack/Coverage.hs:290:14)
2018-04-29 15:29:03.819748: [info] 
An index of the generated HTML coverage reports is available at /Users/dan/scratch/lens-family-th-0.5.0.1/.stack-work/install/x86_64-osx/nightly-2018-04-28/8.4.2/hpc/index.html
@(src/Stack/Coverage.hs:390:9)

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

stack upgrade

The text was updated successfully, but these errors were encountered:

DanBurton · 2018-04-29T22:37:35Z

May be related to the usage of Template Haskell (?)

byorgey · 2019-02-22T20:08:06Z

Bump? I think I am running into this as well. Happy to provide enough details to reproduce if it would be helpful. Like Dan, I have double-checked that as far as I can tell, neither #1008 nor #1411 are the culprit. I am also using Template Haskell (to generate some lenses), though that hardly seems remarkable.

I happened to be on stack 1.7.1 so I upgraded to the latest (1.9.3) to see if that would help; it didn't.

curiousleo · 2020-03-27T10:47:40Z

Bump. I'm running into this too. No Template Haskell involved this time.

Repro:

$ curl -sS https://codeload.github.com/idontgetoutmuch/random/tar.gz/38d38ab65767cee47ed44b0acb8f67e813771d89 | tar xvz
$ cd random-*; stack test --coverage

err0r500 · 2020-05-01T14:20:04Z

I've got the same problem on my project, so I tried with a stack template with same result :

stack new my-project rio
cd my-project
stack test --coverage

output :

Building all executables for `my-project' once. After a successful build of all of them, only specified executables will be rebuilt.
my-project> configure (lib + exe + test)
Configuring my-project-0.1.0.0...
my-project> build (lib + exe + test)
Preprocessing library for my-project-0.1.0.0..
Building library for my-project-0.1.0.0..
[1 of 5] Compiling Paths_my_project
[2 of 5] Compiling Types
[3 of 5] Compiling Import
[4 of 5] Compiling Run
[5 of 5] Compiling Util
Preprocessing executable 'my-project-exe' for my-project-0.1.0.0..
Building executable 'my-project-exe' for my-project-0.1.0.0..
[1 of 2] Compiling Paths_my_project
[2 of 2] Compiling Main
Linking .stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/build/my-project-exe/my-project-exe ...
Preprocessing test suite 'my-project-test' for my-project-0.1.0.0..
Building test suite 'my-project-test' for my-project-0.1.0.0..
[1 of 3] Compiling Paths_my_project
[2 of 3] Compiling UtilSpec
[3 of 3] Compiling Main
Linking .stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/build/my-project-test/my-project-test ...
my-project> copy/register
Installing library in /home/matth/my-project/.stack-work/install/x86_64-linux-tinfo6/0f77fd90bf35fd5f5cebea0aab764d6782b23623e315035c52e134c831e3f74a/8.8.3/lib/x86_64-linux-ghc-8.8.3/my-project-0.1.0.0-EyzZPYII06LGJbZaNa3U4W
Installing executable my-project-exe in /home/matth/my-project/.stack-work/install/x86_64-linux-tinfo6/0f77fd90bf35fd5f5cebea0aab764d6782b23623e315035c52e134c831e3f74a/8.8.3/bin
Registering library for my-project-0.1.0.0..
my-project> test (suite: my-project-test)


Util
  plus2
    basic check
    overflow
    minus 2
      +++ OK, passed 100 tests.

Finished in 0.0013 seconds
3 examples, 0 failures

my-project> Test suite my-project-test passed
Generating coverage report for my-project's test-suite "my-project-test"
Error: The coverage report for my-project's test-suite "my-project-test" did not consider any code. One possible cause
of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.
Completed 2 action(s).
Only one tix file found in /home/matth/my-project/.stack-work/install/x86_64-linux-tinfo6/0f77fd90bf35fd5f5cebea0aab764d6782b23623e315035c52e134c831e3f74a/8.8.3/hpc/, so not generating a unified coverage report.

An index of the generated HTML coverage reports is available at /home/matth/my-project/.stack-work/install/x86_64-linux-tinfo6/0f77fd90bf35fd5f5cebea0aab764d6782b23623e315035c52e134c831e3f74a/8.8.3/hpc/index.html

$ stack --version
Version 2.3.1, Git revision de2a7b694f07de7e6cf17f8c92338c16286b2878 (8103 commits) x86_64 hpack-0.33.0

err0r500 · 2020-05-01T16:01:47Z

Ok, I have been able to get it working :

run the same stack test --coverage (will still fail) then

stack hpc report .

(from your project root)

brandon-leapyear · 2020-07-21T17:34:25Z

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

^ stack hpc report . works.

Interestingly, stack test --coverage works on LTS 14.27, but not LTS 15.0. Probably a breakage in the upgraded GHC version (8.6 -> 8.8)?

brandon-leapyear · 2020-07-29T21:21:44Z

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

Ok so running stack test --coverage -v with LTS 14.27 vs LTS 15.0, I see LTS 14.27 including

--include aeson-schemas-1.2.0-KY8T02FjG98D5Nfn49pCZh:

and LTS 15.0 including

--include "                  aeson-schemas-1.2.0-JXw80ApJdoEEfjMvUrmUow:"

If I manually run the hpc report command that stack is running, removing the whitespace from that include option, it works.

So it seems like the culprit is extraReportArgs in

stack/src/Stack/Coverage.hs

Line 183 in 95aaf2b

: (args ++ extraReportArgs)

which it gets from eincludeName in

stack/src/Stack/Coverage.hs

Lines 138 to 146 in 95aaf2b

    
           case eincludeName of 
        
               Left err -> generateHpcErrorReport reportDir (RIO.display (sanitize (T.unpack err))) 
        
               -- Restrict to just the current library code, if there is a library in the package (see 
        
               -- #634 - this will likely be customizable in the future) 
        
               Right mincludeName -> do 
        
                   let extraArgs = case mincludeName of 
        
                           Just includeNames -> "--include" : intersperse "--include" (map (\n -> n ++ ":") includeNames) 
        
                           Nothing -> [] 
        
                   mreportPath <- generateHpcReportInternal tixSrc reportDir report extraArgs extraArgs

which it gets from findPackageFieldForBuiltPackage in

stack/src/Stack/Coverage.hs

Line 128 in 95aaf2b

    
           eincludeName <- findPackageFieldForBuiltPackage pkgDir (packageIdentifier package) internalLibs hpcNameField

andddddd the bug seems to be in extractField

stack/src/Stack/Coverage.hs

Lines 434 to 438 in 95aaf2b

    
           extractField path = do 
        
               contents <- readFileUtf8 (toFilePath path) 
        
               case asum (map (T.stripPrefix (field <> ": ")) (T.lines contents)) of 
        
                   Just result -> return $ Right result 
        
                   Nothing -> notFoundErr

It seems like Cabal 3 formats the .conf file now.

-- Cabal 2.4
name: aeson-schemas
version: 1.2.0
id: aeson-schemas-1.2.0-KY8T02FjG98D5Nfn49pCZh
key: aeson-schemas-1.2.0-KY8T02FjG98D5Nfn49pCZh

-- Cabal 3
name:                 aeson-schemas
version:              1.2.0
visibility:           public
id:                   aeson-schemas-1.2.0-JXw80ApJdoEEfjMvUrmUow
key:                  aeson-schemas-1.2.0-JXw80ApJdoEEfjMvUrmUow

I'll make a PR for this

Ecosystem changes caused test coverage report generation via `stack test --coverage` to break sometime between lts-13.0 and lts-16.0; this is a workaround. See commercialhaskell/stack#3997

* Bump stack resolver version to 16.0 and update tests - `fail` is being removed from the `Monad` class, so we need to update some constraints. See https://gitlab.haskell.org/ghc/ghc/-/wikis/design/monad-fail - Behavior of getElementCssValue may depend on the version of firefox. - Ecosystem changes caused test coverage report generation via `stack test --coverage` to break sometime between lts-13.0 and lts-16.0; this is a workaround. See commercialhaskell/stack#3997 Co-authored-by: nbloomf <[email protected]>

This was referenced Jul 29, 2020

Fix building coverage report with Cabal 3 #5355

Merged

Upgrade Stack + LTS brandonchinn178/aeson-schemas#40

Merged

snoyberg closed this as completed in #5355 Aug 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stack test --coverage "did not consider any code", fails to produce coverage report #3997

stack test --coverage "did not consider any code", fails to produce coverage report #3997

DanBurton commented Apr 29, 2018

DanBurton commented Apr 29, 2018

byorgey commented Feb 22, 2019 •

edited

Loading

curiousleo commented Mar 27, 2020

err0r500 commented May 1, 2020

err0r500 commented May 1, 2020

brandon-leapyear commented Jul 21, 2020 •

edited

Loading

brandon-leapyear commented Jul 29, 2020 •

edited

Loading

stack test --coverage "did not consider any code", fails to produce coverage report #3997

stack test --coverage "did not consider any code", fails to produce coverage report #3997

Comments

DanBurton commented Apr 29, 2018

Steps to reproduce

Expected

Actual

Stack version

Method of installation

DanBurton commented Apr 29, 2018

byorgey commented Feb 22, 2019 • edited Loading

curiousleo commented Mar 27, 2020

err0r500 commented May 1, 2020

err0r500 commented May 1, 2020

brandon-leapyear commented Jul 21, 2020 • edited Loading

brandon-leapyear commented Jul 29, 2020 • edited Loading

byorgey commented Feb 22, 2019 •

edited

Loading

brandon-leapyear commented Jul 21, 2020 •

edited

Loading

brandon-leapyear commented Jul 29, 2020 •

edited

Loading