Refactor Tests

[svenkeidel]

Dear contributors,

Thanks for putting all this work into this library!

In my opinion, the experience of writing tests and reading test reports is a bit unsatisfying:

• I find it cumbersome to write tests. In particular, the tested property and the test name are at different places (e.g. see HashMapProperties.hs).
• I find it hard to read the test report. I need to scroll through multiple pages of text to find out if one of the test suits failed (I run the test suits with stack test).

Would you consider using a more modern testing framework, such as HSpec, with a friendlier test report? I understand that this might be problematic with backwards compatibility with older GHC versions. If this is not an option, at least consider inlining the test properties so they appear directly below their test name. Let me know what you think.

Regards,
Sven

[sjakobi]

• I find it cumbersome to write tests. In particular, the tested property and the test name are at different places (e.g. see HashMapProperties.hs).

I've made the same experience in other testsuites (mostly tasty-based I believe) and hated that too.

• I find it hard to read the test report. I need to scroll through multiple pages of text to find out if one of the test suits failed (I run the test suits with stack test).

Aren't failures being signaled with the exit code? I.e. a failing test should cause stack test to return a non-zero exit code.

To elide the output for the successful testcases, you can use --hide-successes:

$ stack test --ta --hide-successes  # --ta is short for --test-arguments

The test output is still fairly long since we have 6 separate testsuites! I think we should try merging these into a single one.

Regarding the choice of test framework, I'm mostly familiar with tasty. I like that its maintainer is very responsive to support questions. To address the first issue, we could try using tasty-discover. I've heard good things about hspec too though.

[svenkeidel]

I use hspec in all of my projects. I like it because it has a nice syntax for writing tests and the test reports look good.

The test output is still fairly long since we have 6 separate testsuites! I think we should try merging these into a single one.

Yes, this sounds like a good idea. You can still keep the different test suits in separate files and create a master test suite that imports and combines them all.

[sjakobi]

Test ergonomics should be better now after merging #316 and #351.

• I find it cumbersome to write tests. In particular, the tested property and the test name are at different places (e.g. see HashMapProperties.hs).

We can try to avoid this in the future by eschewing top-level declarations of individual properties, similar to what bytestring does: https://github.com/haskell/bytestring/blob/0aee57e9b705d077a066be04fad01588eef77f2c/tests/Properties/ByteString.hs#L127-L658

We'll have to figure out how to best guide type inference then. -XTypeApplications may help.

[sjakobi]

We can try to avoid this in the future by eschewing top-level declarations of individual properties, similar to what bytestring does: https://github.com/haskell/bytestring/blob/0aee57e9b705d077a066be04fad01588eef77f2c/tests/Properties/ByteString.hs#L127-L658

#453 has introduced this style for the HashMap and HashSet property tests.

This ticket should be mostly fixed now. If there's more to do, please open a new issue! :)