cabal-docspec - another doctest for Haskell
cabal-docspec [OPTION]... [PACKAGE]...
cabal-docspec --no-cabal-plan [OPTION]... [CABALFILE]...
cabal-docspec is a doctest runner closely integrated with cabal-install. In common scenarios cabal-docspec is able to self-configure using cabal-install generated metadata (plan.json). Another important difference is that cabal-docspec doesn't depend on ghc library, but rather invoke the ghci executable.
cabal-docspec doesn't use GHC to parse input files, but rather relies on haskell-lexer for comment extraction. This approach is resilient, but not 100 per cent accurate.
cabal-docspec doesn't interpret library code in ghci, but instead loads precompiled code. The effect is similar as using -fobject-code in GHCi (which is mandatory for packages with FFI, for example). The consequence is that cabal-docspec is unable to evaluate doctest examples in non-exported modules (other-modules), or which use non-exposed symbols.
GHCi is invoked in a package directory, however it is told to not look for modules anywhere (with bare -i flag). This way doctests may use package local files, but the code is not re-intepreted.
In general, all boolean options are enabled with --option and yet again disabled with --no-option. That is, you use the exact same option name but prefix it with "no-". However, in this list we mostly only list and show the --option version of them.
-w, --with-compiler path
: A path to compiler to use to run doctest examples. Must have the same version as in the cabal plan.
--ghc
: Indicate the used compiler is GHC. Currently this options is no-op.
--cabal-plan : Read plan.json produced by cabal-install to find out project packages and their dependencies. When turned (with --no-cabal-plan), paths to the cabal files have to be given. Also only the global package db is considered for dependencies. Default --cabal-plan.
--preserve-it
: Preserve it variable, i.e. the result in of previous expression. Default --no-preserve-it.
--strip-comments
: Strip Haskell comments from examples and the outputs. Especially outputs are assumed to be Haskell-like. Default --no-strip-comments.
-Z, --ignore-trailing-space
: Strip trailing whitespace from the produced outputs. Default --no-ignore-trailing-space.
--setup expr
: An additional expression to execute as setup for all examples. Can be specified multiple times.
--extra-package pkgname
: An extra package to make available in GHCi session. The package must exist in the plan. Sublibrary syntax mypkg:mysublib is also accepted.
--timeout seconds
: Timeout for evaluation of single expression. Default: 3 seconds. Long timeouts may allow GHCi to acquire a lot of resources. On the other hand, too short timeout may cause false negatives.
--timeout-message message
: Message to return when the evaluation is timed out. Default is * Hangs forever *.
--ghci-rtsopts options
: RTS options for GHCi process
--check-properties
: Evaluate prop> expr x y using quickCheck (expr x y). Requires QuickCheck package in the plan. Default --no-check-properties.
--property-variables varlist
: Variables to quantify over in properties.
-X extension
: Language extension to start GHCi session with. Can be specified multiple times.
-I directory
: Add directory to the directory search list for #include files.
--phase1
: Stop after the first phase. First phase consists of discovering the modules, and extracting the doctest examples from their comments.
--phase2
: Stop after the second phase, i.e. evaluation in GHCi phase.
-m, --module modulename
: Check only these modules. Default is to check all.
--builddir dir
: Directory to look for plan.json and local package database.
-v, --verbose
: Increase verbosity level. Can be specified multiple times.
-q, --quiet
: Decrease verbosity level. Can be specified multiple times.
--version
: Display numeric version.
--help
: Display short help message.
--man
: Display this manual.
It' is possible to provide cabal-docspec configuration through fields in a .cabal file.
x-docspec-options: [OPTION]...
: These options will be applied before command line options, and allow configuration of cabal-docspec per component under test.
x-docspec-extra-packages: [PKG]...
: A (space separated) list of extra packages. See --extra-package.
x-docspec-property-variables: [VAR]...
: A (space separated) list of property variables. See --property-variables.
For most packages it is sufficient to run cabal-docspec after cabal v2-build all:
cabal v2-build all
cabal-docspec
The GHC source tree doesn't have cabal-install generated plan.json, therefore we use --no-cabal-plan and supply the libraries/base/base.cabal path. There are some examples using explanatory comments, --strip-comments makes them work. Some examples are illustrating non-termination, therefore short --timeout is justified. Yet, it has to be long enough so the terminating examples have time to run. We also set RTS options, reducing the maximum stack to make stack overflow exceptions occur earlier. Since the examples below use symbols from the mtl, deepseq and bytestring packages, we make them available. Finally, some modules are documented with no-Prelude assumption, therefore we have to turn it off.
cabal-docspec \
-w $PWD/_build/stage1/bin/ghc \
-I $PWD/includes \
--no-cabal-plan \
--strip-comments \
--timeout 2 \
--ghci-rtsopts "-K500K" \
--extra-package=mtl --extra-package=deepseq --extra-package=bytestring \
-XNoImplicitPrelude \
libraries/base/base.cabal
The lens library uses simple-reflect library for illustration of some examples. However, simple-reflect is not a dependency of lens library. One way to have add such dependency is to create dummy test-suite with it.
test-suite doctests
type: exitcode-stdio-1.0
main-is: doctests.hs
hs-source-dirs: tests
default-language: Haskell2010
build-depends: base, simple-reflect >= 0.3.1
Where doctests.hs doesn't need to do anything in particular, for example it could be:
module Main where
main :: IO ()
main = do
putStrLn "This test-suite exists only to add dependencies"
putStrLn "To run doctests: "
putStrLn " cabal build all --enable-tests"
putStrLn " cabal-docspec"
The bare cabal-docspec command works, because needed extra packages are configured using x-docspec-extra-packages field in a package definition library stanza:
library
...
x-docspec-extra-packages: simple-reflect
NOTE: This section is edited version of a part of the Doctest README.markdown. cabal-docspec reuses the way examples are specified.
Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.
module Fib where
-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)A comment line starting with >>> denotes an expression.
All comment lines following an expression denote the result of that expression.
Result is defined by what a REPL (e.g. ghci) prints to stdout and stderr when evaluating that expression.
Examples from a single Haddock comment are grouped together and share the same scope. E.g. the following works:
-- |
-- >>> let x = 23
-- >>> x + 42
-- 65If an example fails, subsequent examples from the same group are skipped. E.g. for
-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print nprint n is not tried, because let n = x + y fails (y is not in scope!).
Because cabal-docspec uses compiled library, calling :reload: after each group doesn't cause performance problems. For that reason, cabal-docspec doesn't have --fast variant, it is not needed.
You can put setup code in a named chunk with the name $setup. The setup code is run before each example group. If the setup code produces any errors/failures, all tests from that module are skipped.
Here is an example:
module Foo where
import Bar.Baz
-- $setup
-- >>> let x = 23 :: Int
-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42GHCi supports commands which span multiple lines, and the same syntax works for Doctest:
-- |
-- >>> :{
-- let
-- x = 1
-- y = 2
-- in x + y + multiline
-- :}
-- 6
multiline = 3Note that >>> can be left off for the lines following the first: this is so that haddock does not strip leading whitespace. The expected output has whitespace stripped relative to the :}.
Some peculiarities on the ghci side mean that whitespace at the very start is lost. This breaks the example broken` since the x and y aren't aligned from ghci's perspective. A workaround is to avoid leading space, or add a newline such that the indentation does not matter:
{- | >>> :{
let x = 1
y = 2
in x + y + works
:}
6
-}
works = 3
{- | >>> :{
let x = 1
y = 2
in x + y + broken
:}
3
-}
broken = 3If there are no blank lines in the output, multiple lines are handled automatically.
-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!If however the output contains blank lines, they must be noted explicitly with . For example,
import Data.List ( intercalate )
-- | Double-space a paragraph.
--
-- Examples:
--
-- >>> let s1 = "\"Every one of whom?\""
-- >>> let s2 = "\"Every one of whom do you think?\""
-- >>> let s3 = "\"I haven't any idea.\""
-- >>> let paragraph = unlines [s1,s2,s3]
-- >>> putStrLn $ doubleSpace paragraph
-- "Every one of whom?"
-- <BLANKLINE>
-- "Every one of whom do you think?"
-- <BLANKLINE>
-- "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . linesAny lines containing only three dots (...) will match one or more lines with arbitrary content. For instance,
-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- bazIf a line contains three dots and additional content, the three dots will match anything within that line:
-- |
-- >>> putStrLn "foo bar baz"
-- foo ... bazHaddock (since version 2.13.0) has markup support for properties cabal-docspec can verify properties with QuickCheck. Note: this works somewhat differently than it does in Doctest.
By default properties are not checked. cabal-docspec has a simple mechanism to evaluate properties enabled by --check-properties. For it to work, the QuickCheck package has to be in the install plan.
A simple property looks like this:
-- |
-- prop> \xs -> sort xs == (sort . sort) (xs :: [Int])The lambda abstraction is required by default. cabal-docspec will quantify over variables passed in with --property-variables command line flag.
With --property-variables xs the following will work:
-- |
-- prop> sort xs == (sort . sort) (xs :: [Int])Doctest uses a hack to find which variables are free in the the expression. cabal-docspec's approach is more deterministic, as it doesn't try to infer anything.
Also, in contrast to Doctest, cabal-docspec doesn't use the polyQuickCheck trick. Therefore some false properties may pass
quickCheck $ \xs -> reverse xs === xs
+++ OK, passed 100 tests.That property passes because the list element type defaults to (). To avoid defaulting you may override the default class resolution in a $setup block
-- $setup
-- >>> default (Integer, Double)Then the property above will fail:
quickCheck $ \xs -> reverse xs === xs
*** Failed! Falsified (after 4 tests and 4 shrinks):
[1,0]
[0,1] /= [1,0]A complete example that uses setup code is below:
module Fib where
-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary
-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)You can put examples into named chunks, and not refer to them in the export list. That way they will not be part of the generated Haddock documentation, but cabal-docspec will still find them.
-- $
-- >>> 1 + 1
-- 2There's two sets of GHC extensions involved when running Doctest:
- The set of GHC extensions that are active when compiling the module code.
- The set of GHC extensions that are active when executing the Doctest examples. (These are not influenced by the LANGUAGE pragmas in the file.)
Unlike Doctest, cabal-docspec doesn't compile libraries, therefore you don't need to do anything special for the first point.
The recommended way to enable extensions for cabal-docspec examples is to specify them as -X flags. Because set of enabled extensions persist even after :reload, it is better to embrace that fact and enable them globally.
Another way to enable extensions, which is compatible with Doctest, is to switch them on like this:
-- |
-- >>> :set -XTupleSections
-- >>> fst' $ (1,) 2
-- 1
fst' :: (a, b) -> a
fst' = fstAll warnings are enabled by default.
-Wmultiple-module-files
: Found multiple files matching the exposed module.
-Wmissing-module-file
: No files found matching a module. For example modules which are preprocessed (.hsc etc).
-Wtimeout
: Evaluation of an expression timed out.
-Wunknown-extension
: Warn if extension passed via -X seems to be unknown. The known extension list is from Cabal library.
-Winvalid-field
: Warn when parsing of cabal package file fields fails.
-Wcpphs
: C preprocessor (cpphs) warnings.
-Werror-in-setup
: There was an error in evaluting $setup.
-Wskipped-property
: Warn about properties when --skip-properties (the default) is enabled.
Properties (prop>) are recognized but not evaluated.
Literate Haskell is not supported.
Dependencies' install-includes folders are not added to C preprocess search path.
GHC-7.0 relies that Char type is in scope. This is an implementation artifact.
cabal-docspec tests library documentation from the outside. It doesn't even try to look into an implementation for some secret bits, only to find examples. In this sense it is more principled (than Doctest). Therefore you might need to repeat imports in a $setup block. OTOH, the implementation's imports never interfere with doctests.
Named documentaton chunks are the only possibly hidden part of source text, which cabal-docspec uses.
One way is to redefine the symbol in a $setup block using a qualified module name.
let null = Module.Under.Test.nullThis way it will shadow both Prelude.null and Module.Under.Test.null, and ambiguous symbol errors won't appear.
Another option is to use -XNoImplicitPrelude and import Prelude explicitly.
cabal-docspec reads a plan.json file, which is generated by cabal-install. That file contains (almost) all required information for cabal-docspec to invoke ghci with the correct arguments.
It is generated by cabal-install as a side-effect of running the solver. For example even
cabal build --dry-run
is enough. However, without libraries actually being built, cabal-docspec won't work.
No, cabal-doctest doesn't need one.
The library code is loaded as pre-compiled object code, not interpreted code. As a result, the :reload command doesn't force code to be re-interpreted each time, making to cheap to run.
pre-compiled object, the :reload command is cheap.
It doesn't cause the re-interpretation of the sources.
Yes.
No. The test-suite is there to ensure that the extra dependencies are built by cabal-install. We can also use a dummy package for that purpose, but a test-suite is more lightweight.
As an alternative to this approach, with cabal-install-3.4 you may use
extra-packages: simple-reflect
in the cabal.project file.
In general, no. As long as the library and extra dependencies used by doctests are built, cabal-docspec shold work fine.
There are a few differences.
-
The same cabal-docspec binary works with all GHC versions. Also with versions which don't have .ghc.environment file feature.
-
cabal-docspec doesn't interpret the source code. Though, Doctest could have that mode too.
-
Because cabal-docspec uses plan.json information, it doesn't have problems with the visibility of packages. For example Prelude.Compat from base-compat and base-compat-batteries won't cause ambiguous module problems, as long as the library being tested itself depends only on either one.
cabal-docspec can only test the exported interfaces, so it's not possible to test other-modules. However, cabal-docspec does test internal libraries. Therefore you can put the internal modules into internal library and then cabal-docspec will be able to test them.
doctest(1) https://hackage.haskell.org/package/doctest
https://github.com/phadej/cabal-extras
Copyright © 2020-2023 Oleg Grenrus. License GPLv2-or-later: GNU GPL version 2 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Written by Oleg Grenrus. Doctest comment extraction and comparison functions are originally from Doctest by Simon Hengel. Cpphs is written by Malcolm Wallace. Other dependencies are written by their respective authors.