stack "errors" when building package with benchmark that depends on itself

[RyanGlScott]

Originally discovered here. Steps to reproduce:

$ stack unpack vector-binary-instances
Unpacked vector-binary-instances-0.2.3.2 to /home/ryanglscott/Documents/Hacking/Haskell/vector-binary-instances-0.2.3.2/
$ cd vector-binary-instances-0.2.3.2/
$ stack init --resolver lts-5.16
Using cabal packages:
- vector-binary-instances.cabal

Selected resolver: lts-5.16
Initialising configuration using resolver: lts-5.16
Total number of user packages considered: 1
Writing configuration to file: stack.yaml
All done.
$ stack bench --only-dependencies
...
$ stack bench                                                                             
vector-binary-instances-0.2.3.2: benchmarks                                               
vector-binary-instances-0.2.3.2: configure (bench)                                        
setup-Simple-Cabal-1.22.5.0-ghc-7.10.3: No benchmarks enabled. Did you                    
remember to configure with '--enable-benchmarks'?                                         
Configuring vector-binary-instances-0.2.3.2...                                            
vector-binary-instances-0.2.3.2: build (bench)
Preprocessing library vector-binary-instances-0.2.3.2...
[1 of 1] Compiling Data.Vector.Binary ( Data/Vector/Binary.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.5.0/build/Data/Vector/Binary.o )
In-place registering vector-binary-instances-0.2.3.2...
Preprocessing benchmark 'benchmarks' for vector-binary-instances-0.2.3.2...
[1 of 1] Compiling Main             ( benchmarks/Benchmarks.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.5.0/build/benchmarks/benchmarks-tmp/Main.o )
Linking .stack-work/dist/x86_64-linux/Cabal-1.22.5.0/build/benchmarks/benchmarks ...
Completed 2 action(s).

--  While building package vector-binary-instances-0.2.3.2 using:
      /home/ryanglscott/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.22.5.0-ghc-7.10.3 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.22.5.0 bench benchmarks
    Process exited with code: ExitFailure 1

stack seems to be confused about vector-binary-instances's benchmarks, for the reason that (I assume) its benchmarks transitively depends on vector-binary-instances itself. It still appears to build correctly, but it exits with an error code and doesn't actually run the benchmarks.

[mgsloan]

Oh, hmm, I think this is because cabal does not support cyclic benchmarks (though I think in theory it does on cabal master / perhaps the recent release). We just need to implement benchmark running rather than delegating to cabal.

[mgsloan]

As unpleasant as this is, setting the priority at P2 because there's a direct work-around: stack exec my-bench-name. You can build without having it exit with failure via stack build :my-bench-name --no-run-benchmarks or stack build --bench --no-run-benchmarks

[sjakobi]

@mgsloan:

As unpleasant as this is, setting the priority at P2 because there's a direct work-around: stack exec my-bench-name. You can build without having it exit with failure via stack build :my-bench-name --no-run-benchmarks or stack build --bench --no-run-benchmarks

That doesn't seem to work because my-bench-name isn't on the PATH. Is there a better workaround than digging into .stack-work to find the benchmark executable?

[mgsloan]

Yeah, I think it should be possible for us to do our own benchmark running rather than delegating to Cabal.

[mgsloan]

Finally figured this out! When there's a benchmark / test with cyclic deps, it wasn't waiting for the build to complete before attempting to run.