testpaths = A A/B doesn't test folder A (used to work in v7.4.4)

[Famlam]

In the project Osmose we have the following configuration:

[pytest]
# Note that some files are excluded in "conftest.py"
norecursedirs = *
testpaths = . plugins plugins/tests modules analysers
python_files = *.py
python_classes = Test
python_functions = test test*

# Suppress warnings from shapely.affinity, which are fixed on a latter version
filterwarnings =
  ignore:.*invalid escape sequence.*:DeprecationWarning

Note especially the testpaths = . plugins plugins/tests modules analysers line together with norecursedirs = *

In version 7.4.4 this meant:

• Check the main folder
• Check the plugins folder
• Check the plugins/tests folder
• Check the modules folder
• Check the analysers folder

In version 8.2.2 however this skips the files in the plugins folder, e.g.

• Check the main folder
• Check the plugins folder
• Check the plugins/tests folder
• Check the modules folder
• Check the analysers folder

The header line when running the pytest command still mentions testpaths: ., plugins, plugins/tests, modules, analysers

============================= test session starts ==============================
platform linux -- Python 3.11.[9](https://github.com/osm-fr/osmose-backend/actions/runs/9844396084/job/27177682389#step:10:10), pytest-8.2.2, pluggy-1.5.0
rootdir: /__w/osmose-backend/osmose-backend
configfile: pytest.ini
testpaths: ., plugins, plugins/tests, modules, analysers
plugins: pylama-8.4.1
collected 68 items

I therefore think this change in behavior is likely not intended.

The directory structure, simplified, is:

root
|- analysers
| |- subfolders (to skip)
|- modules
| |- subfolders (to skip)
|- other folders
| |- subfolders (to skip)
|- plugins
| |- tests
| |- other subfolders (to skip)

Environment:

• Pytest: 8.2.2 (also confirmed to fail on 8.0.2)
• Operating System: Debian GNU/Linux 11 (bullseye)

pip list

Package                   Version
------------------------- ---------------
antlr4-python3-runtime    4.13.1
anyio                     4.4.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     23.2.0
Babel                     2.15.0
beautifulsoup4            4.12.3
bleach                    6.1.0
branca                    0.7.2
certifi                   2024.7.4
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
comm                      0.2.2
debugpy                   1.8.2
decorator                 5.1.1
defusedxml                0.7.1
exceptiongroup            1.2.2
executing                 2.0.1
fastjsonschema            2.20.0
fqdn                      1.5.1
future                    1.0.0
geojson                   3.1.0
h11                       0.14.0
httpcore                  1.0.5
httpx                     0.27.0
idna                      3.7
importlib-metadata        8.0.0
iniconfig                 2.0.0
ipykernel                 6.29.5
ipyleaflet                0.19.1
ipython                   8.18.1
ipywidgets                8.1.3
isoduration               20.11.0
jedi                      0.19.1
jinja2                    3.1.4
json5                     0.9.25
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
jupyter-client            8.6.2
jupyter-core              5.7.2
jupyter-events            0.10.0
jupyter-leaflet           0.19.1
jupyter-lsp               2.2.5
jupyter-server            2.14.2
jupyter-server-terminals  0.5.3
jupyterlab                4.2.3
jupyterlab-pygments       0.3.0
jupyterlab-server         2.27.2
jupyterlab-widgets        3.0.11
libarchive                0.4.8
lockfile                  0.12.2
mapbox-vector-tile        1.2.1
MarkupSafe                2.1.5
matplotlib-inline         0.1.7
mccabe                    0.7.0
mistune                   3.0.2
mypy                      1.10.1
mypy-extensions           1.0.0
nbclient                  0.10.0
nbconvert                 7.16.4
nbformat                  5.10.4
nest-asyncio              1.6.0
nose2                     0.15.1
notebook                  7.2.1
notebook-shim             0.2.4
numpy                     2.0.0
osmium                    3.7.0
overrides                 7.7.0
packaging                 24.1
pandas                    2.2.2
pandocfilters             1.5.1
parso                     0.8.4
pexpect                   4.9.0
pip                       20.3.4
pkg-resources             0.0.0
platformdirs              4.2.2
pluggy                    1.5.0
polib                     1.2.0
prometheus-client         0.20.0
prompt-toolkit            3.0.47
protobuf                  3.20.3
psutil                    6.0.0
psycopg2-binary           2.9.9
ptyprocess                0.7.0
pure-eval                 0.2.2
pyclipper                 1.3.0.post5
pycodestyle               2.12.0
pycparser                 2.22
pydocstyle                6.3.0
pyflakes                  3.2.0
pygments                  2.18.0
PyKOpeningHours           1.6.0
pylama                    8.4.1
pyproj                    3.6.1
pytest                    8.2.2
python-dateutil           2.9.0.post0
python-json-logger        2.0.7
pytz                      2024.1
PyYAML                    6.0.1
pyzmq                     26.0.3
referencing               0.35.1
regex                     2024.5.15
requests                  2.32.3
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.19.0
scipy                     1.13.1
Send2Trash                1.8.3
sentry-sdk                2.9.0
setuptools                44.1.1
shapely                   2.0.5
six                       1.16.0
sniffio                   1.3.1
snowballstemmer           2.2.0
soupsieve                 2.5
stack-data                0.6.3
termcolor                 2.4.0
terminado                 0.18.1
tiletanic                 1.1.0
tinycss2                  1.3.0
tomli                     2.0.1
tornado                   6.4.1
traitlets                 5.14.3
traittypes                0.2.1
transporthours            0.0.4
types-certifi             2021.10.8.3
types-pkg-resources       0.1.3
types-python-dateutil     2.9.0.20240316
types-requests            2.32.0.20240712
types-termcolor           1.1.6.2
types-toml                0.10.8.20240310
typing-extensions         4.12.2
tzdata                    2024.1
Unidecode                 1.3.8
uri-template              1.3.0
urllib3                   2.2.2
vt2geojson                0.2.1
wcwidth                   0.2.13
webcolors                 24.6.0
webencodings              0.5.1
websocket-client          1.8.0
wheel                     0.34.2
widgetsnbextension        4.0.11
xmltodict                 0.13.0
xyzservices               2024.6.0
zipp                      3.19.2

Note that if I remove either plugins or plugins/tests from testpaths, everything behaves as expected (except that the removed directory isn't checked of course)

Related issue: osm-fr/osmose-backend#2266

[RonnyPfannschmidt]

@bluetech my first rough idea for verification would be that the culling of test-path parent directories for the Directory collection may be overzealous if one of the parent folders is path of the testdirs

it should be gracefully handled, and perhaps warn that the extra subdir entry is of no consequence
we also should note that completely disabling recursion is currently a edge case we haven't worked with

[potiuk]

We stumbled upon the same issue in Airflow - we have a way to make selective checks (based on what has changed in the incoming commit we select folders to run tests on (but we want to recursively search them).

In some cases the selective checks resulted in (roughly):

pytest tests/providers tests/providers/amazon tests/providers/google

In pytest 7.* that collected all tests/providers tests recursively (including many other subfolders of tests/providers - including amazon, google tests recursively as well). However with Pytest 8.3.2 that we have currently this only collects "amazon" and "google".

I downgraded pytest and checked that (as expected) the behaviour changed in Pytest 8.0.0.

I workarounded it in apache/airflow#41499 - (I remove provider subfolders from the list) when I see I have the parent "tests/providers" also added), however, I think that behaviour is a bit unexpected - I guess it is a side effect of one of the changes in https://docs.pytest.org/en/stable/changelog.html#collection-changes, but it might be pretty disruptive if somoene had similar case - because it does not give an indication that only a subset of tests are run.

Possibly restoring old behaviour should be considered if possible.