PyTest 3.7 duplicating tests during collection

[DanielSchiavini]

After upgrading to pytest 3.7, our projects suddently are collecting each test twice.
All tests run 2 times and the total percentage in the end of the run is between 33% and 55%.

I managed to create a small project to reproduce the issue:
testpytest.zip

Empty files:
./__init__.py
./lib/__init__.py
./lib/tests/__init__.py
./tests/__init__.py

Test files:
./lib/tests/TestPyTest.py
./tests/TestPyTest.py

from unittest import TestCase
class TestPyTest(TestCase):
    def test(self):
        pass

./pytest.ini

[pytest]
testpaths = .
python_files = Test*.py
python_classes = Test
python_functions = test

./Dockerfile

FROM python:3.7 AS base
WORKDIR /usr/src/app

FROM base AS pytest36
RUN pip install pytest==3.6.4
ADD . .
RUN ENV_NAME=testing pytest

FROM base AS pytest37
RUN pip install pytest==3.7
ADD . .
RUN ENV_NAME=testing pytest

Output of docker build .:

[...]
Step 4/10 : RUN pip install pytest==3.6.4
 ---> Using cache
 ---> 31c440fd1976
Step 5/10 : ADD . .
 ---> aa1d7f89c771
Step 6/10 : RUN ENV_NAME=testing pytest
 ---> Running in 72af6d63f895
============================= test session starts ==============================
platform linux -- Python 3.7.0, pytest-3.6.4, py-1.5.4, pluggy-0.7.1
rootdir: /usr/src/app, inifile: pytest.ini
collected 2 items

lib/tests/TestPyTest.py .                                                [ 50%]
tests/TestPyTest.py .                                                    [100%]

=========================== 2 passed in 0.04 seconds ===========================
Removing intermediate container 72af6d63f895
 ---> d843066e44ea
Step 7/10 : FROM base AS pytest37
 ---> 22d8eec60ce2
Step 8/10 : RUN pip install pytest==3.7
 ---> Using cache
 ---> d05ea16fa3f3
Step 9/10 : ADD . .
 ---> a8b18ec3669b
Step 10/10 : RUN ENV_NAME=testing pytest
 ---> Running in 9d5b48e1beb0
============================= test session starts ==============================
platform linux -- Python 3.7.0, pytest-3.7.0, py-1.5.4, pluggy-0.7.1
rootdir: /usr/src/app, inifile: pytest.ini
collected 3 items

lib/tests/TestPyTest.py ..                                               [ 33%]
tests/TestPyTest.py .

=========================== 3 passed in 0.04 seconds ===========================
Removing intermediate container 9d5b48e1beb0
 ---> 1affcfd67144
Successfully built 1affcfd67144
Successfully tagged pytest:latest

Note how pytest 3.6.4 correctly finds 2 tests, while pytest 3.7 collects 3 tests and reports a total of 33% run.

[nicoddemus]

Thanks @DanielSchiavini for the report.

I did some investigation and found that <Package 'lib'> and <Package 'lib/tests'> both are collecting <Module 'TestPyTest.py'> from lib/tests/TestPyTest.py... I would need more time to investigate than I have right now, I'm afraid.

[jonozzz]

I looked into this and the problem is in the Package.collect() code. I based my logic on a bad assumption that __init__.py would always come first when iterating through filesystem. That's so wrong and this issue just proves it. Ugh!
Working on a fix now.

[cdeil]

I'm getting this error, which might be the same cause as the issue mentioned above?

$ pytest -v astropy/table
/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
Internet access disabled
=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.7.0, pytest-3.7.1, py-1.5.4, pluggy-0.7.1 -- /Users/deil/software/anaconda3/envs/gammapy-dev-37/bin/python
cachedir: .pytest_cache

Running tests with Astropy version 3.1.dev22380.
Running tests in astropy/table.

Date: 2018-08-24T08:58:29

Platform: Darwin-17.7.0-x86_64-i386-64bit

Executable: /Users/deil/software/anaconda3/envs/gammapy-dev-37/bin/python

Full Python Version: 
3.7.0 | packaged by conda-forge | (default, Aug  7 2018, 14:28:25) 
[Clang 6.1.0 (clang-602.0.53)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.15.0
Scipy: 1.1.0
Matplotlib: 2.2.3
h5py: not available
Pandas: not available
Cython: 0.28.5
astropy_helpers: 3.0.2
Using Astropy options: remote_data: none.

rootdir: /Users/deil/work/code/astropy, inifile: setup.cfg
plugins: remotedata-0.3.0, openfiles-0.3.0, doctestplus-0.1.3, arraydiff-0.2
collected 0 items / 1 errors                                                                                                                                                                              

================================================================================================= ERRORS ==================================================================================================
_______________________________________________________________________________ ERROR collecting astropy/table/__init__.py ________________________________________________________________________________
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/hooks.py:258: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/manager.py:67: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/manager.py:61: in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/_pytest/python.py:210: in pytest_collect_file
    return ihook.pytest_pycollect_makemodule(path=path, parent=parent)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/hooks.py:258: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/manager.py:67: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/pluggy/manager.py:61: in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/_pytest/python.py:215: in pytest_pycollect_makemodule
    return Package(path, parent)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/_pytest/python.py:552: in __init__
    self, fspath, parent=parent, config=config, session=session, nodeid=nodeid
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/_pytest/nodes.py:351: in __init__
    fspath = py.path.local(fspath)  # xxx only for test_resultlog.py?
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/py/_path/local.py:158: in __init__
    self.strpath = abspath(path)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/posixpath.py:372: in abspath
    if not isabs(path):
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/posixpath.py:67: in isabs
    sep = _get_sep(s)
../../../software/anaconda3/envs/gammapy-dev-37/lib/python3.7/posixpath.py:42: in _get_sep
    if isinstance(path, bytes):
E   RecursionError: maximum recursion depth exceeded while calling a Python object
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 error in 21.05 seconds =========================================================================================

You should be able to reproduce via:

conda create --name trouble python=3.7 numpy cython pytest
conda activate trouble
git clone https://github.com/astropy/astropy.git
cd astropy
pytest -v astropy/table

Let me know if you can't reproduce, or if I should file a separate issue for this.

[nicoddemus]

Hi @cdeil,

I could not reproduce your issue on Ubuntu 18, latest conda and a fresh clone:

 1472 passed, 10 skipped, 11 xfailed, 5 error in 58.53 seconds

All the errors I got was the error about tree fixture being called directly:

    @pytest.fixture
    def bst():
>       return tree()
E       _pytest.deprecated.RemovedInPytest4Warning: Fixture tree called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

The fix for that by the way is to change bst to:

    @pytest.fixture
    def bst(tree):
        return tree

Either way could you please open a new issue? It doesn't seem to be the issue as the one described here.

[nicoddemus]

For that fix I've opened astropy/astropy#7769 already. 😁

[DanielSchiavini]

Thanks for picking this up!