Run test suites without rebuilding executables

[lostman]

Running stack test also builds all executables. Using fast-tags package as an example :

❯ stack test        
fast-tags-1.1.0: configure (test)
Configuring fast-tags-1.1.0...
fast-tags-1.1.0: build (test)
Building fast-tags-1.1.0...
Preprocessing library fast-tags-1.1.0...
In-place registering fast-tags-1.1.0...
Preprocessing executable 'fast-tags' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-osx/Cabal-1.18.1.5/build/fast-tags/fast-tags ...
Preprocessing test suite 'test-fast-tags' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-osx/Cabal-1.18.1.5/build/test-fast-tags/test-fast-tags ...
fast-tags-1.1.0: test (suite: test-fast-tags)
tests
  tokenize
    "a::b->c":                                                                OK (0.02s)
(...)

I can simulate the desired behavior by defining a testonly flag and marking all executables with if flag(testonly) buildable: False. With executables:

time stack test
stack test  5.79s user 1.60s system 127% cpu 5.777 total

And without:

time stack test --flag fast-tags:test
stack test --flag fast-tags:test  4.73s user 1.45s system 130% cpu 4.724 total

These are incremental, not clean, builds. Skipping executables makes running tests slightly faster. This is particularly useful when there are many executables and only a single test-suite.

[snoyberg]

I don't believe runghc Setup configure has an option to disable executable building unfortunately, though it would be nice. Instead, you could try using components so force only some things to build, though I haven't tested this, e.g. stack test packagename:test:testname.

[lostman]

@snoyberg I tried this. It is equivalent to stack test. Moreover, if there are many test suites picking a specific one doesn't work:

❯ stack test fast-tags:test-fast-tags
fast-tags-1.1.0: build (test)
Building fast-tags-1.1.0...
Preprocessing library fast-tags-1.1.0...
In-place registering fast-tags-1.1.0...
Preprocessing executable 'fast-tags' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-osx/Cabal-1.18.1.5/build/fast-tags/fast-tags ...
Preprocessing test suite 'test-fast-tags' for fast-tags-1.1.0...
Linking
.stack-work/dist/x86_64-osx/Cabal-1.18.1.5/build/test-fast-tags/test-fast-tags
...
Preprocessing test suite 'test-fast-tags2' for fast-tags-1.1.0...
Linking
.stack-work/dist/x86_64-osx/Cabal-1.18.1.5/build/test-fast-tags2/test-fast-tags2
...
fast-tags-1.1.0: test (suite: test-fast-tags)
tests
  tokenize
    "a::b->c":                                                                OK
(...)
All 250 tests passed (0.06s)
fast-tags-1.1.0: test (suite: test-fast-tags2)
tests
  tokenize
    "a::b->c":                                                                OK
(...)
All 250 tests passed (0.05s)

Here I defined a duplicate test-fast-tags2 test suite.

[lostman]

Also, with cabal it is possible to do something like this:

cabal install --only-dependencies --enable-tests
cabal build my-test-suite
./dist/.../my-test-suite

Not ideal, but does the trick. I don't think there's a way to do the same with stack:

❯ stack build fast-tags:test-fast-tags                      
fast-tags-1.1.0: build
Setup.hs: Cannot build the test suite 'test-fast-tags' because test suites are
not enabled. Run configure with the flag --enable-tests

[snoyberg]

I confirmed that in fact it was not working previously. I've just pushed a commit that should address it. Can you give it a shot?

[lostman]

👍

Works great. Thanks!

Only the first test suite:

(master ✗) fast-tags stack test fast-tags:test-fast-tags
fast-tags-1.1.0: configure (test)
Configuring fast-tags-1.1.0...
fast-tags-1.1.0: build (test)
Preprocessing library fast-tags-1.1.0...
In-place registering fast-tags-1.1.0...
Preprocessing test suite 'test-fast-tags' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/test-fast-tags/test-fast-tags ...
fast-tags-1.1.0: test (suite: test-fast-tags)
tests
(...)

Only the second:

(master ✗) fast-tags stack test fast-tags:test-fast-tags2
fast-tags-1.1.0: configure (test)
Configuring fast-tags-1.1.0...
fast-tags-1.1.0: build (test)
Preprocessing library fast-tags-1.1.0...
In-place registering fast-tags-1.1.0...
Preprocessing test suite 'test-fast-tags2' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/test-fast-tags2/test-fast-tags2 ...
fast-tags-1.1.0: test (suite: test-fast-tags2)
tests
(...)

And both:

(master ✗) fast-tags stack test
fast-tags-1.1.0: configure (test)
Configuring fast-tags-1.1.0...
fast-tags-1.1.0: build (test)
Preprocessing library fast-tags-1.1.0...
In-place registering fast-tags-1.1.0...
Preprocessing test suite 'test-fast-tags' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/test-fast-tags/test-fast-tags ...
Preprocessing test suite 'test-fast-tags2' for fast-tags-1.1.0...
Linking .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/test-fast-tags2/test-fast-tags2 ...
fast-tags-1.1.0: test (suite: test-fast-tags)
tests
(...)

[snoyberg]

Cool, thanks for reporting and confirming!