prove

This project was originally called 'CL-TEST-MORE'.

'prove' is yet another unit testing framework for Common Lisp.

The advantages of 'prove' are:

• Various simple functions for testing and informative error messages
• ASDF integration
• Extensible test reporters
• Colorizes the report if it's available (note for SLIME)
• Reports test durations

Quickstart

1. Writing a test file

(in-package :cl-user)
(defpackage my-test
  (:use :cl
        :prove))
(in-package :my-test)

(plan 3)

(ok (not (find 4 '(1 2 3))))
(is 4 4)
(isnt 1 #\1)

(finalize)

2. Run a test file

(prove:run #P"myapp/tests/my-test.lisp")
(prove:run #P"myapp/tests/my-test.lisp" :reporter :list)

See also: ASDF integration, Reporters

3. Get a report

Installation

You can install 'prove' via Quicklisp.

(ql:quickload :prove)

Testing functions

(ok test &optional desc)

Checks if test is true (non-NIL).

(ok 1)
;->  ✓ 1 is expected to be T

(is got expected &rest test-args)

Checks if got is equivalent to expected.

(is 1 1)
;->  ✓ 1 is expected to be 1

(is #(1 2 3) #(1 2 3))
;->  × #(1 2 3) is expected to be #(1 2 3)

(is #(1 2 3) #(1 2 3) :test #'equalp)
;->  ✓ #(1 2 3) is expected to be #(1 2 3)

;; with description
(is 1 #\1 "Integer = Character ?")
;->  × Integer = Character ?

(isnt got expected &rest test-args)

Checks if got is not equivalent to expected.

(isnt 1 1)
;->  × 1 is not expected to be 1

(isnt #(1 2 3) #(1 2 3))
;->  ✓ #(1 2 3) is not expected to be #(1 2 3)

(is-values got expected &rest test-args)

Checks if the multiple values of got is equivalent to expected. This is same to (is (multiple-value-list got) expected).

(defvar *person* (make-hash-table))

(is-values (gethash :name *person*) '("Eitaro" T))
;->  × (NIL NIL) is expected to be ("Eitaro" T)

(setf (gethash :name *person*) "Eitaro")

(is-values (gethash :name *person*) '("Eitaro" T))
;->  ✓ ("Eitaro" T) is expected to be ("Eitaro" T)

(is-type got expected-type &optional desc)

Checks if got is a type of expected-type.

(is-type #(1 2 3) 'simple-vector)
;->  ✓ #(1 2 3) is expected to be a type of SIMPLE-VECTOR (got (SIMPLE-VECTOR 3))

(is-type (make-array 0 :adjustable t) 'simple-vector)
;->  × #() is expected to be a type of SIMPLE-VECTOR (got (VECTOR T 0))

(like got regex &optional desc)

Checks if got matches a regular expression regex.

(like "Hatsune 39" "\\d")
;->  ✓ "Hatsune 39" is expected to be like "\\d"

(like "初音ミク" "\\d")
;->  × "初音ミク" is expected to be like "\\d"

(is-print got expected &optional desc)

Checks if got outputs expected to *standard-output*

(is-print (princ "Hi, there") "Hi, there")
;->  ✓ (PRINC "Hi, there") is expected to output "Hi, there" (got "Hi, there")

(is-error form condition &optional desc)

Checks if form raises a condition and that is a subtype of condition.

(is-error (error "Something wrong") 'simple-error)
;->  ✓ (ERROR "Something wrong") is expected to raise a condition SIMPLE-ERROR (got #<SIMPLE-ERROR "Something wrong" {100628FE53}>)

(define-condition my-error (simple-error) ())

(is-error (error "Something wrong") 'my-error)
;->  × (ERROR "Something wrong") is expected to raise a condition MY-ERROR (got #<SIMPLE-ERROR "Something wrong" {100648E553}>)

(is-expand got expected &optional desc)

Checks if got will be macroexpanded to expected.

(is-expand (when T (princ "Hi")) (if T (progn (princ "Hi"))))
;->  ✓ (WHEN T (PRINC "Hi")) is expected to be expanded to (IF T
;                                                          (PROGN (PRINC "Hi"))) (got (IF T
;                                                                                         (PROGN
;                                                                                          (PRINC
;                                                                                           "Hi"))
;                                                                                         NIL))

If a symbol that starts with "$" is contained, it will be treated as a gensym.

(pass desc)

This will always be passed. This is convenient if the test case is complicated and hard to test with ok.

(pass "Looks good")
;->  ✓ Looks good

(fail desc)

This will always be failed. This is convenient if the test case is complicated and hard to test with ok.

(fail "Hopeless")
;->  × Hopeless

(skip how-many why)

Skip a number of how-many tests and mark them passed.

(skip 3 "No need to test these on Mac OS X")
;->  ✓ No need to test these on Mac OS X (Skipped)
;    ✓ No need to test these on Mac OS X (Skipped)
;    ✓ No need to test these on Mac OS X (Skipped)

(subtest desc &body body)

Run tests of body in a new sub test suite.

(subtest "Testing integers"
  (is 1 1)
  (is-type 1 'bit)
  (is-type 10 'fixnum))
;->      ✓ 1 is expected to be 1
;        ✓ 1 is expected to be a type of BIT (got BIT)
;        ✓ 10 is expected to be a type of FIXNUM (got (INTEGER 0 4611686018427387903))
;->  ✓ Testing integers

Other functions

(diag desc)

Outputs desc to a *test-result-output*.

(diag "Gonna run tests")
;-> # Gonna run tests

(plan num)

Declares a number of num tests are going to run. If finalize is called with no plan, a warning message will be output. num is allows to be NIL if you have no plan yet.

(finalize)

Finalizes the current test suite and outputs the test reports.

(slow-threshold milliseconds)

Set the threshold of slow test durations for the current test suite. The default threshold value is prove:*default-slow-threshold*.

(slow-threshold 150)

Reporters

You can change the test report formats by setting prove:*default-reporter* to :list, :dot, :tap or :fiveam. The default value is :list.

prove:run also takes a keyword argument :reporter.

List (Default)

The :list repoter outputs test results list as test cases pass or fail.

Dot

The :dot reporter outputs a series of dots that represent test cases, failures highlight in red, skipping in cyan.

FiveAM

The :fiveam reporter outputs test results like FiveAM does.

TAP

The :tap reporter outputs in Test Anything Protocol format.

Tips

Debugging with CL debugger

Set prove:*debug-on-error* T for invoking CL debugger whenever getting an error during running tests.

Colorize test reports on SLIME

SLIME doesn't support to color with ANSI colors in the REPL buffer officially.

You can add the feature by using slime-repl-ansi-color.el.

After installing it, set prove:*enable-colors* to T before running tests.

;; A part of my ~/.sbclrc
(ql:quickload :prove)
(setf prove:*enable-colors* t)

The following snippet is a little bit complicated, however it would be better if you don't like to load prove in all sessions.

(defmethod asdf:perform :after ((op asdf:load-op) (c (eql (asdf:find-system :prove))))
  (setf (symbol-value (intern (string :*enable-colors*) :prove)) t))

ASDF integration

Add :defsystem-depends-on (:prove-asdf) to your testing ASDF system to enable :test-file in the :components.

:test-file is same as :file except it will be loaded only when asdf:test-system.

;; Main ASDF system
(defsystem my-app

  ;; ...

  :in-order-to ((test-op (test-op my-app-test))))

;; Testing ASDF system
(defsystem my-app-test
  :depends-on (:my-app
               :prove)
  :defsystem-depends-on (:prove-asdf)
  :components
  ((:test-file "my-app"))
  :perform (test-op :after (op c)
                    (funcall (intern #.(string :run) :prove) c)))

To run tests, execute asdf:test-system or prove:run in your REPL.

(asdf:test-system :my-app)
(asdf:test-system :my-app-test)

;; Same as 'asdf:test-system' except it returns T or NIL as the result of tests.
(prove:run :my-app-test)

Changing default test function

Test functions like is uses prove:*default-test-function* for testing if no :test argument is specified. The default value is #'equal.

Changing output stream

Test reports will be output to prove:*test-result-output*. The default value is T, which means *standard-output*.

Running tests on Travis CI

Although Common Lisp isn't supported by Travis CI officially, you can run tests by using cl-travis.

Here's a list of .travis.yml from projects using prove on Travis CI:

• Clack
• CL-DBI
• Woo
• fast-http
• defclass-std

Bugs

Please report any bugs to [email protected], or post an issue to GitHub.

License

Copyright (c) 2010-2014 Eitaro Fukamachi <[email protected]>
'prove' and CL-TEST-MORE is freely distributable under the MIT License (http://www.opensource.org/licenses/mit-license).