Package tests fail in current environment

[olebole]

Since a few days, pytesting the package fails in our CI tests:

$ pytest-3
================================== test session starts ==================================
platform linux -- Python 3.8.3, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ole/Projects/2011/debian/python/pytest/filter-subpackage/.hypothesis/examples')
Matplotlib: 3.2.1
Freetype: 2.10.1
rootdir: /home/ole/Projects/2011/debian/python/pytest/filter-subpackage, inifile: setup.cfg, testpaths: tests, pytest_filter_subpackage
plugins: asdf-2.6.0, hypothesis-4.36.2, remotedata-0.3.2, mpl-0.11, openfiles-0.5.0, astropy-header-0.1.2, doctestplus-0.6.1, arraydiff-0.3
collected 7 items                                                                       

tests/test_filter_subpackage.py .FFFFFF                                           [100%]

======================================= FAILURES ========================================
_____________________________________ test_with_rst _____________________________________

testdir = <Testdir local('/tmp/pytest-of-ole/pytest-3/test_with_rst0')>
testpackage = None

    def test_with_rst(testdir, testpackage):
        # Make sure all rst tests are also picked up when using pytest-doctestplus
        testdir.makeini("""
        [pytest]
        doctest_plus = enabled
        """)
        reprec = testdir.inline_run('--doctest-rst')
>       reprec.assertoutcome(passed=8, failed=3)

/home/ole/Projects/2011/debian/python/pytest/filter-subpackage/tests/test_filter_subpackage.py:17: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.pytester.HookRecorder object at 0x7f5432be0c70>, passed = 8, skipped = 0
failed = 3

    def assertoutcome(self, passed=0, skipped=0, failed=0):
        realpassed, realskipped, realfailed = self.listoutcomes()
>       assert passed == len(realpassed)
E       AssertionError: assert 8 == 7
E        +  where 7 = len([<TestReport 'docs/a/index.rst::index.rst' when='call' outcome='passed'>, <TestReport 'docs/c/index.rst::index.rst' wh...n='call' outcome='passed'>, <TestReport 'testpackage/c/d/tests/test_d.py::test_d_1' when='call' outcome='passed'>, ...])

/usr/lib/python3/dist-packages/_pytest/pytester.py:317: AssertionError
--------------------------------- Captured stdout call ----------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ole/Projects/2011/debian/python/pytest/filter-subpackage/.hypothesis/examples')
Matplotlib: 3.2.1
Freetype: 2.10.1
rootdir: /tmp/pytest-of-ole/pytest-3/test_with_rst0, inifile: tox.ini
plugins: asdf-2.6.0, hypothesis-4.36.2, remotedata-0.3.2, mpl-0.11, openfiles-0.5.0, astropy-header-0.1.2, doctestplus-0.6.1, arraydiff-0.3
collected 10 items

docs/a/index.rst .                                                       [ 10%]
docs/c/index.rst .                                                       [ 20%]
testpackage/a/tests/test_a.py ..                                         [ 40%]
testpackage/b/tests/test_b.py .F                                         [ 60%]
testpackage/c/d/tests/test_d.py .F                                       [ 80%]
testpackage/c/tests/test_c.py .F                                         [100%]

=================================== FAILURES ===================================
___________________________________ test_b_2 ___________________________________

    def test_b_2():
>       assert 'b' == 'a'
E       AssertionError: assert 'b' == 'a'
E         - b
E         + a

testpackage/b/tests/test_b.py:6: AssertionError
___________________________________ test_d_2 ___________________________________

    def test_d_2():
>       assert 'd' == 'a'
E       AssertionError: assert 'd' == 'a'
E         - d
E         + a

testpackage/c/d/tests/test_d.py:6: AssertionError
___________________________________ test_c_2 ___________________________________

    def test_c_2():
>       assert 'c' == 'a'
E       AssertionError: assert 'c' == 'a'
E         - c
E         + a

testpackage/c/tests/test_c.py:6: AssertionError
====================== 3 failed, 7 passed in 0.07 seconds ======================

Only the first failure is shown above, the full log is here.
There may be a relation to an update of pytest-doctestplus from version 0.5.0 to version 0.6.1; at least that is one difference between the successfull and the failing run.

[bnavigator]

Did doctestplus < 0.6.1 not count correctly? Because the docs in the testdir only have two examples to doctest, which makes the expected count of succeeding tests equal 7, not 8. Just change to passed=7.

[Hellseher]

Hi,

A gentle ping on this issue. Is it still actual?

Regards.

[bnavigator]

Yes.

[pllim]

Dropping Python 3.8 in #21 . FYI