This package contains a plugin for the pytest framework that provides advanced doctest support and enables the testing of reStructuredText (".rst") files. It was originally part of the astropy core package, but has been moved to a separate package in order to be of more general use.
This plugin provides advanced features for testing example Python code that is included in Python docstrings and in standalone documentation files.
Good documentation for developers contains example code. This is true of both standalone documentation and of documentation that is integrated with the code itself. Python provides a mechanism for testing code snippets that are provided in Python docstrings. The unit test framework pytest provides a mechanism for running doctests against both docstrings in source code and in standalone documentation files.
This plugin augments the functionality provided by Python and pytest by providing the following features:
- approximate floating point comparison for doctests that produce floating point results (see Floating Point Comparison)
- skipping particular classes, methods, and functions when running doctests (see Skipping Tests)
- handling doctests that use remote data in conjunction with the pytest-remotedata plugin (see Remote Data)
- optional inclusion of
*.rst
files for doctests (see Setup and Configuration)
The pytest-doctestplus
plugin can be installed using pip
:
$ pip install pytest-doctestplus
It is also possible to install the latest development version from the source repository:
$ git clone https://github.com/astropy/pytest-doctestplus $ cd pytest-doctestplus $ pip install .
In either case, the plugin will automatically be registered for use with
pytest
.
This plugin provides two command line options: --doctest-plus
for enabling
the advanced features mentioned above, and --doctest-rst
for including
*.rst
files in doctest collection.
This plugin can also be enabled by default by adding doctest_plus = enabled
to the [tool:pytest]
section of the package's setup.cfg
file.
The plugin is applied to all directories and files that pytest
collects.
This means that configuring testpaths
and norecursedirs
in
setup.cfg
also affects the files that will be discovered by
pytest-doctestplus
. In addition, this plugin provides a
doctest_norecursedirs
configuration variable that indicates directories
that should be ignored by pytest-doctestplus
but do not need to be ignored
by other pytest
features.
Using pytest
's built-in --doctest-modules
option will override the
behavior of this plugin, even if doctest_plus = enabled
in setup.cfg
,
and will cause the default doctest plugin to be used. However, if for some
reason both --doctest-modules
and --doctest-plus
are given, the
pytest-doctestplus
plugin will be used, regardless of the contents of
setup.cfg
.
This plugin respects the doctest options that are used by the built-in doctest
plugin and are set in doctest_optionflags
in setup.cfg
. By default,
ELLIPSIS
and NORMALIZE_WHITESPACE
are used. For a description of all
doctest settings, see the doctest documentation.
The pytest-doctestplus
plugin defines doctest directives that are used
to control the behavior of particular features. For general information on
directives and how they are used, consult the documentation. The specifics
of the directives that this plugin defines are described in the sections below.
You can use testsetup
and testcleanup
in Sphinx RST to run code that is
not visible in rendered document. However, due to how pytest-doctestplus
works, the code within needs to be prepended by >>>
. For example:
.. testsetup:: >>> x = 42 .. testcleanup:: >>> del x
Some doctests may produce output that contains string representations of floating point values. Floating point representations are often not exact and contain roundoffs in their least significant digits. Depending on the platform the tests are being run on (different Python versions, different OS, etc.) the exact number of digits shown can differ. Because doctests work by comparing strings this can cause such tests to fail.
To address this issue, the pytest-doctestplus
plugin provides support for a
FLOAT_CMP
flag that can be used with doctests. For example:
>>> 1.0 / 3.0 # doctest: +FLOAT_CMP
0.333333333333333311
When this flag is used, the expected and actual outputs are both parsed to find any floating point values in the strings. Those are then converted to actual Python float objects and compared numerically. This means that small differences in representation of roundoff digits will be ignored by the doctest. The values are otherwise compared exactly, so more significant (albeit possibly small) differences will still be caught by these tests.
This flag can be enabled globally by adding it to setup.cfg
as in
doctest_optionflags =
NORMALIZE_WHITESPACE
ELLIPSIS
FLOAT_CMP
If code in a doctest emits a warning and you want to make sure that warning is silenced,
you can make use of the IGNORE_WARNINGS
flag. For example:
>>> import numpy as np
>>> np.mean([]) # doctest: +IGNORE_WARNINGS
np.nan
If code in a doctest emits a warning and you want to make sure that warning is
shown, you can make use of the SHOW_WARNINGS
flag. This is useful when
warnings are turned into errors by pytest, and also because by default warnings
are printed to stderr. This is the opposite from IGNORE_WARNINGS
so
obviously the two flags should not be used together. For example:
>>> import numpy as np
>>> np.mean([]) # doctest: +SHOW_WARNINGS
RuntimeWarning: Mean of empty slice.
RuntimeWarning: invalid value encountered in double_scalars
np.nan
Doctest provides the +SKIP
directive for skipping statements that should
not be executed when testing documentation.
>>> open('file.txt') # doctest: +SKIP
In Sphinx .rst
documentation, whole code example blocks can be skipped with the
directive
.. doctest-skip::
>>> import asdf
>>> asdf.open('file.asdf')
However, it is often useful to be able to skip docstrings associated with particular functions, methods, classes, or even entire files.
The pytest-doctestplus
plugin provides a way to indicate that certain
docstrings should be skipped altogether. This is configured by defining the
variable __doctest_skip__
in each module where tests should be skipped. The
value of __doctest_skip__
should be a list of wildcard patterns for all
functions/classes whose doctests should be skipped. For example:
__doctest_skip__ = ['myfunction', 'MyClass', 'MyClass.*']
skips the doctests in a function called myfunction
, the doctest for a
class called MyClass
, and all methods of MyClass
.
Module docstrings may contain doctests as well. To skip the module-level doctests:
__doctest_skip__ = ['.', 'myfunction', 'MyClass']
To skip all doctests in a module:
__doctest_skip__ = ['*']
It is also possible to skip certain doctests depending on whether particular
dependencies are available. This is configured by defining the variable
__doctest_requires__
at the module level. The value of this variable is
a dictionary that indicates the modules that are required to run the doctests
associated with particular functions, classes, and methods.
The keys in the dictionary are wildcard patterns like those described above, or tuples of wildcard patterns, indicating which docstrings should be skipped. The values in the dictionary are lists of module names that are required in order for the given doctests to be executed.
Consider the following example:
__doctest_requires__ = {('func1', 'func2'): ['scipy']}
Having this module-level variable will require scipy
to be importable
in order to run the doctests for functions func1
and func2
in that
module.
Similarly, in Sphinx .rst
documentation, whole code example blocks can be
conditionally skipped if a dependency is not available.
.. doctest-requires:: asdf
>>> import asdf
>>> asdf.open('file.asdf')
Finally, it is possible to skip collecting doctests in entire subpackages by
using the doctest_subpackage_requires
in the [tool:pytest]
section of
the package's setup.cfg
file. The syntax for this option is a list of
path = requirements
, e.g.:
doctest_subpackage_requires = astropy/wcs/* = scipy>2.0;numpy>1.14 astropy/cosmology/* = scipy>1.0
Multiple requirements can be specified if separated by semicolons.
The pytest-doctestplus
plugin can be used in conjunction with the
pytest-remotedata plugin in order to control doctest code that requires
access to data from the internet. In order to make use of these features, the
pytest-remotedata
plugin must be installed, and remote data access must
be enabled using the --remote-data
command line option to pytest
. See
the pytest-remotedata plugin documentation for more details.
The following example illustrates how a doctest that uses remote data should be marked:
>>> from urlib.request import urlopen
>>> url = urlopen('http://astropy.org') # doctest: +REMOTE_DATA
The +REMOTE_DATA
directive indicates that the marked statement should only
be executed if the --remote-data
option is given. By default, all
statements marked with the remote data directive will be skipped.
Whole code example blocks can also be marked to control access to data from the internet this way:
.. doctest-remote-data::
>>> import requests
>>> r = requests.get('https://www.astropy.org')
To use the additional directives when building your documentation with sphinx
you may want to enable the sphinx extension which registers these directives
with sphinx. Doing so ensures that sphinx correctly ignores these directives,
running the doctests with sphinx is not supported. To do this, add
'pytest_doctestplus.sphinx.doctestplus'
to your extensions
list in your
conf.py
file.
Questions, bug reports, and feature requests can be submitted on github.
This plugin is licensed under a 3-clause BSD style license - see the
LICENSE.rst
file.