Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doctest plugin runs when unwanted #4954

Open
cjw296 opened this issue Mar 19, 2019 · 17 comments
Open

doctest plugin runs when unwanted #4954

cjw296 opened this issue Mar 19, 2019 · 17 comments
Labels
plugin: doctests related to the doctests builtin plugin type: bug problem that needs to be addressed

Comments

@cjw296
Copy link

cjw296 commented Mar 19, 2019

This file fails when run explicitly with pytest:

$ pytest --doctest-glob *.nope README.rst 
...
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
...
...______ [doctest] README.rst __
093         'banner': Required(str),
...
102 >>> print(open('/etc/my_app/config.yaml').read())
UNEXPECTED EXCEPTION: FileNotFoundError(2, 'No such file or directory')

Things work fine if I just pytest in the root directory. The problem is that the doctest plugin is kicking in when it has not been requested, and indeed even when I try and explicitly stop it from working on this file. I use sybil for testing examples in my documentation as it provides for more flexibility.

Here's my full environment:

$ pip freeze
alabaster==0.7.12
atomicwrites==1.3.0
attrs==19.1.0
Babel==2.6.0
bleach==3.1.0
certifi==2019.3.9
chardet==3.0.4
-e [email protected]:Simplistix/configurator.git@1090b4db182490e523ca9b7415fd87ba784e25ae#egg=configurator
coverage==4.5.3
docutils==0.14
idna==2.8
imagesize==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
mock==2.0.0
more-itertools==6.0.0
packaging==19.0
pbr==5.1.3
pkginfo==1.5.0.1
pluggy==0.9.0
py==1.8.0
pyfakefs==3.5.8
Pygments==2.3.1
pyparsing==2.3.1
pytest==4.3.1
pytest-cov==2.6.1
pytz==2018.9
PyYAML==5.1
readme-renderer==24.0
requests==2.21.0
requests-toolbelt==0.9.1
setuptools-git==1.2
six==1.12.0
snowballstemmer==1.2.1
Sphinx==1.8.5
sphinxcontrib-websupport==1.1.0
sybil==1.0.9
testfixtures==6.6.1
toml==0.10.0
tqdm==4.31.1
twine==1.13.0
urllib3==1.24.1
voluptuous==0.11.5
webencodings==0.5.1
@blueyed
Copy link
Contributor

blueyed commented Mar 20, 2019

pytest --doctest-glob *.nope README.rst

Haven't investigate, but make sure that *.nope does not get expanded by your shell already, i.e. quote it.

@blueyed
Copy link
Contributor

blueyed commented Mar 20, 2019

Related code: https://github.com/blueyed/pytest/blob/033849a31ec7d2b9f0493fa51c77289eab657af1/src/_pytest/doctest.py#L104-L107.
Added in acd286f initially.

The glob pattern is ignored for txt and rst files given on the command line explicitly.

Why do you specify README.rst explicitly?

@blueyed blueyed added type: question general question, might be closed after 2 weeks of inactivity plugin: doctests related to the doctests builtin plugin labels Mar 20, 2019
@cjw296
Copy link
Author

cjw296 commented Mar 20, 2019

I was very sure that *.nope did not match any files ;-)

"The glob pattern is ignored for txt and rst files given on the command line explicitly." feels like a bug to me. Why specify README.rst explicitly? Same reason people normally specify a subset of their tests, I just wanted to run those tests, not the whole unit test suite.

@blueyed
Copy link
Contributor

blueyed commented Mar 20, 2019

Yeah.. but if you specify "README.rst" it runs the (doc)tests from it.

It is like withtestpaths but specifying a file explicitly.

@cjw296
Copy link
Author

cjw296 commented Mar 20, 2019

It evidently does, but why should it? it doesn't when I don't otherwise limit discovery.
If I do, pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)

To be clear: this is far from a show stopper, but it is a bit of a wart and it would be great to find a way to fix it!

@blueyed
Copy link
Contributor

blueyed commented Mar 20, 2019

If I do, pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)

But pytest t-foo.py ignores testpaths also.

@nicoddemus
Copy link
Member

If you pass a file explicitly on the command-line, pytest will run that file as a test, regardless of test file pattern configuration or testpaths. For example if you have tests on foo.py, running just pytest . won't pick up foo.py, but pytest foo.py will. The same happens for doctests as @blueyed pointed out in the code.

I think the rationale is that if you are explicitly passing the file in the command-line, you are telling pytest to run tests from that file.

Not sure what were the requirements that lead to this, but I don't think we should change this without a good reason because this will certainly break some workflows out there.

@cjw296
Copy link
Author

cjw296 commented Mar 29, 2019

[fixed this comment up as I realised I was misreading some output]

Sorry, but this doesn't feel right. -vv output gives some more detail: it looks like the file is being processed twice, once with the stuff I've configured in conftest.py and then additionally, erroneously, as a doctest:

$ pytest -vvx README.rst 
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 14 items                                                                                                                                                    

README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [  7%]
README.rst::line:45,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 14%]
README.rst::line:66,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 21%]
README.rst::line:92,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 28%]
README.rst::line:107,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 35%]
README.rst::line:108,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 42%]
README.rst::line:112,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 50%]
README.rst::line:114,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 57%]
README.rst::line:116,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 64%]
README.rst::line:122,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 71%]
README.rst::line:124,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 78%]
README.rst::line:126,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 85%]
README.rst::line:128,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 92%]
README.rst::README.rst <- /Users/chris/vcs/git/configurator/README.rst FAILED                                                                                   [100%]

Now, if I specify an item within the file, things work as expected:

$ pytest -vvx README.rst::line:26,column:1
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 1 item                                                                                                                                                      

README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [100%]

====================================================================== 1 passed in 0.08 seconds =======================================================================

The problem is that these tests shared state within a module, they're testing the examples within documentation using Sybil, so I really want to be able to run a single file without the annoying and erroneous doctest stuff kicking in.

How do I do that?

@blueyed
Copy link
Contributor

blueyed commented Mar 29, 2019

once with the stuff I've configured in conftest.py

What is that?
Do you mean setup.cfg/pytest.ini maybe instead?

I suggest digging into the code maybe yourself (I've left a pointer already).

(btw: with 4.4 it would also display "testpaths: …" in the header, I wonder if that covers this case also?)

@cjw296
Copy link
Author

cjw296 commented Mar 29, 2019

No, I mean conftest.py.
I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.

@blueyed
Copy link
Contributor

blueyed commented Mar 29, 2019

I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.

It is not really, but I just wondered if files for doctests would show up there (like they should).

@cjw296
Copy link
Author

cjw296 commented Mar 29, 2019

Just to be clear: I'm not using doctests and I don't want to be using doctests. I want pytest to stop erroneously treating my non-doctest .rst files as doctests, which it only does if I specify a particular file on the command line.

My currently workaround is to do:

pytest -k README.rst

That works fine, so there's no massive priority on fixing this, but it feels like you're trying to tell me it's not a bug, in which case I will respectfully disagree ;-)

@blueyed
Copy link
Contributor

blueyed commented Mar 29, 2019

Try -p no:doctest then maybe (might require pytest 4.4 / current features branch though).

@cjw296
Copy link
Author

cjw296 commented Mar 29, 2019

I already have a workaround thanks...

@blueyed
Copy link
Contributor

blueyed commented Mar 29, 2019

And I agree that the collection logic in the doctest plugin is a bit wonky.

As for your workaround: it will collect all files, and then filter based on a pattern, which is slower than collecting just one file.

@blueyed blueyed added type: bug problem that needs to be addressed and removed type: question general question, might be closed after 2 weeks of inactivity labels Mar 29, 2019
@cjw296
Copy link
Author

cjw296 commented Dec 2, 2020

Just to note that this is still an issue in pytest 6, but that putting addopts = -p no:doctest in the config file does stop the pytest doctest plugin from getting in the way.

@nicoddemus
Copy link
Member

Thanks @cjw296,

Not sure how we could change the defaults so they don't break existing workflows though. Any ideas? If so I suggest we open a new "proposal" with a plan to deprecate automatically running doctests in .rst and .txt files, but in a way that won't get users by surprise (for example, someone upgrades pytest to a version which no longer runs those files as doctests, and doesn't notice).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plugin: doctests related to the doctests builtin plugin type: bug problem that needs to be addressed
Projects
None yet
Development

No branches or pull requests

3 participants