pipenv + Appveyor: pytest can't find plugins

[hoefling]

Hi, I have encountered a following issue when playing with pipenv in my private projects. When issuing pipenv run pytest, no plugins are found and the test execution fails sooner or later (either the args in addopts, or plugin fixtures are not recognized).

I can reproduce this issue on Windows (Appveyor) only, running the builds on Travis succeeds, also the issue is not reproducible on any of my Linux machines (CentOS 7 / Fedora 27 / Ubuntu 16.04 / Debian Jessie) or MacOS High Sierra.

I prepared a small repository that reproduces this issue: pytest-pipenv-sandbox. Here's an example of failing build on Appveyor.

I'm not sure whether this is an issue with pipenv or pytest; I'm fine with rejecting this issue, I will open another one for pipenv then.

[pytestbot]

GitMate.io thinks possibly related issues are #1479 (pytest silently stops importing plugins on ImportError), #1978 (pytest not finding any tests under module), #613 (Pytest can't find locally installed projects), #2196 (Pytest on cygwin doesn't find files), and #1652 (plugins resolved by pip instead of core pytest packages).

[RonnyPfannschmidt]

at first glance i beleive this is a pipenv issue - pytest uses setuptools entrypoints to find the plugins

[nicoddemus]

Hi @hoefling,

Thanks for providing the sample repository. I've tried it on Windows but it works for me:

(.env36) λ set PIPENV_IGNORE_VIRTUALENVS=1

X:\pytest-pipenv-sandbox    master± 
(.env36) λ pipenv run pytest
Creating a virtualenv for this project…
Using x:\pytest-pipenv-sandbox\.env36\scripts\python.exe (3.6.3) to create virtualenv…
Already using interpreter x:\pytest-pipenv-sandbox\.env36\scripts\python.exe
Using base prefix 'C:\\Users\\Bruno\\AppData\\Local\\Programs\\Python\\Python36'
New python executable in x:\.virtualenvs\pytest-pipenv-sandbox--6bpSMGi\Scripts\python.exe
Installing setuptools, pip, wheel...done.

Virtualenv location: x:\.virtualenvs\pytest-pipenv-sandbox--6bpSMGi
======================================= test session starts =======================================
platform win32 -- Python 3.6.3, pytest-3.5.0, py-1.5.3, pluggy-0.6.0 -- D:\Miniconda3\python.exe
cachedir: .pytest_cache
rootdir: X:\pytest-pipenv-sandbox, inifile: pytest.ini
plugins: cov-2.5.1, mock-1.9.0
collected 1 item

tests/test_spam.py::test_eggs PASSED                                                         [100%]

------------------- generated xml file: X:\pytest-pipenv-sandbox\unittests.xml --------------------

----------- coverage: platform win32, python 3.6.3-final-0 -----------
Name               Stmts   Miss  Cover
--------------------------------------
spam\__init__.py       0      0   100%
Coverage XML written to file coverage.xml

==================================== 1 passed in 0.03 seconds =====================================

Can you please provide the full output of your run? (Note that I'm running from inside a virtualenv with the PIPENV_IGNORE_VIRTUALENVS set, this might be relevant)

[hoefling]

Hi, sorry for the delayed response. I tackled the issue once again and found out that the root of all evil is pipenv not installing the dependencies specified with environment markers. Why this happens, I have no idea atm, but this is surely not something pytest should care about. What happens is the following:

• pytest has an env marker colorama; sys.platform=="win32"
• colorama is not being installed
• the plugins are loaded by pluggy
• pluggy tries to load the plugin entrypoint
• the DistributionNotFound is raised, but swallowed, so no plugin is registered

This has the effect that although plugin packages are installed, pytest does not see them on test run and fails. The interesting point is that pytest does not complain about colorama missing and runs the tests just fine until the first plugin usage. If I install colorama explicitly, the tests run with no errors.

Thus, if you don't mind, I'd propose the issue solved as unrelated to pytest. Sorry for the noise and thanks for keeping up the great work!

[j9ac9k]

I'm having a similar thing going on with pytest-qt plugin. I'm able to verify that it's installed as before I do the test I do a piping run pip list

Expand AppVeyor Log

Build started
git clone -q https://github.com/j9ac9k/timeview.git C:\projects\timeview
git fetch -q origin +refs/pull/4/merge:
git checkout -qf FETCH_HEAD
Running Install scripts
if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod ` https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | ` Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { ` throw "There are newer queued builds for this pull request, failing early." }
SET PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
python --version
Python 3.6.6
python -c "import struct; print(struct.calcsize('P') * 8)"
64
python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0
pip install pipenv --upgrade-strategy=only-if-needed
Collecting pipenv
  Downloading https://files.pythonhosted.org/packages/eb/64/9b2747d54f2008ac3dfe86c0b1c8ec126042726fd8a540d5208d26732701/pipenv-2018.7.1-py3-none-any.whl (5.0MB)
Collecting certifi (from pipenv)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
Requirement already satisfied: pip>=9.0.1 in c:\python36-x64\lib\site-packages (from pipenv) (18.0)
Collecting virtualenv-clone>=0.2.5 (from pipenv)
  Downloading https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=36.2.1 in c:\python36-x64\lib\site-packages (from pipenv) (39.0.1)
Requirement already satisfied: virtualenv in c:\python36-x64\lib\site-packages (from pipenv) (15.1.0)
Installing collected packages: certifi, virtualenv-clone, pipenv
Successfully installed certifi-2018.4.16 pipenv-2018.7.1 virtualenv-clone-0.3.0
pipenv --python=3.6 install --dev .
Creating a virtualenv for this project...
Pipfile: C:\projects\timeview\Pipfile
Using C:\Python36-x64\python.exe (3.6.6) to create virtualenv...
Running virtualenv with interpreter C:\Python36-x64\python.exe
Using base prefix 'C:\\Python36-x64'
New python executable in C:\Users\appveyor\.virtualenvs\timeview-aDISNJXo\Scripts\python.exe
Installing setuptools, pip, wheel...done.
Setting project for timeview-aDISNJXo to C:\projects\timeview
Virtualenv location: C:\Users\appveyor\.virtualenvs\timeview-aDISNJXo
Installing dependencies from Pipfile.lock (9f16e2)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
c:\python36-x64\lib\site-packages\setuptools\pep425tags.py:89: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
c:\python36-x64\lib\site-packages\setuptools\pep425tags.py:93: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
c:\python36-x64\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
pipenv run pip list
Package           Version   Location            
----------------- --------- --------------------
appdirs           1.4.3     
aspy.yaml         1.1.1     
atomicwrites      1.1.5     
attrs             18.1.0    
black             18.6b4    
bleach            2.1.3     
cached-property   1.4.3     
certifi           2018.4.16 
cffi              1.11.5    
cfgv              1.1.0     
chardet           3.0.4     
click             6.7       
cmarkgfm          0.4.2     
CommonMark        0.5.4     
docutils          0.14      
flake8            3.5.0     
flake8-bugbear    18.2.0    
flake8-mypy       17.8.0    
future            0.16.0    
html5lib          1.0.1     
identify          1.1.4     
idna              2.7       
llvmlite          0.24.0    
mccabe            0.6.1     
more-itertools    4.2.0     
mypy              0.620     
nodeenv           1.3.2     
numba             0.39.0    
numpy             1.15.0    
pip               18.0      
pkginfo           1.4.2     
pluggy            0.6.0     
pre-commit        1.10.4    
py                1.5.4     
pycodestyle       2.3.1     
pycparser         2.18      
pyEDFlib          0.1.13    
pyflakes          1.6.0     
Pygments          2.2.0     
PyQt5             5.11.2    
PyQt5-sip         4.19.12   
pyqtgraph         0.10.0    
pytest            3.6.3     
pytest-qt         3.0.0     
pytest-runner     4.2       
PyYAML            4.2b4     
QtAwesome         0.4.4     
QtPy              1.4.2     
readme-renderer   21.0      
recommonmark      0.4.0     
requests          2.19.1    
requests-toolbelt 0.8.0     
scipy             1.1.0     
setuptools        40.0.0    
six               1.11.0    
SQLAlchemy        1.2.10    
timeview          0.1.1     c:\projects\timeview
toml              0.9.4     
tqdm              4.24.0    
twine             1.11.0    
typed-ast         1.1.0     
urllib3           1.23      
virtualenv        16.0.0    
webencodings      0.5.1     
wheel             0.31.1    
pipenv run python -m pytest -v ./tests
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 -- C:\Users\appveyor\.virtualenvs\timeview-aDISNJXo\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\projects\timeview, inifile: setup.cfg
collecting ... collected 4 items
tests/func/testBasicGUI.py::testViewer ERROR                             [ 25%]
tests/func/testBasicGUI.py::testMakeWaveform ERROR                       [ 50%]
tests/unit/testProcessors.py::TestProcessors::test_F0Analyzer PASSED     [ 75%]
tests/unit/testProcessors.py::TestProcessors::test_PeakTracker PASSED    [100%]
=================================== ERRORS ====================================
________________________ ERROR at setup of testViewer _________________________
file C:\projects\timeview\tests\func\testBasicGUI.py, line 9
  def testViewer(viewer: Viewer, qtbot):
file C:\projects\timeview\tests\func\conftest.py, line 12
  @pytest.fixture(scope="function")
  def viewer(qtbot) -> Viewer:
E       fixture 'qtbot' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_xml_attribute, record_xml_property, recwarn, tmpdir, tmpdir_factory, viewer, viewerWithExample
>       use 'pytest --fixtures [testpath]' for help on them.
C:\projects\timeview\tests\func\conftest.py:12
_____________________ ERROR at setup of testMakeWaveform ______________________
file C:\projects\timeview\tests\func\testBasicGUI.py, line 19
  def testMakeWaveform(viewerWithExample: Viewer, qtbot):
file C:\projects\timeview\tests\func\conftest.py, line 21
  @pytest.fixture(scope="function")
  def viewerWithExample(viewer: Viewer, qtbot) -> Viewer:
file C:\projects\timeview\tests\func\conftest.py, line 12
  @pytest.fixture(scope="function")
  def viewer(qtbot) -> Viewer:
E       fixture 'qtbot' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_xml_attribute, record_xml_property, recwarn, tmpdir, tmpdir_factory, viewer, viewerWithExample
>       use 'pytest --fixtures [testpath]' for help on them.
C:\projects\timeview\tests\func\conftest.py:12
====================== 2 passed, 2 error in 4.74 seconds ======================

appveyor.yml

environment:
  global:
    PYTHON: "C:\\Python36-x64"
    PLATFORM: x64
    PYTEST_QT_API: "pyqt5"

install:
  - ps: if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod `
      https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | `
      Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { `
        throw "There are newer queued builds for this pull request, failing early." }

  # Prepend newly installed Python to the PATH of this build (this cannot be
  # done from inside the powershell script as it would require to restart
  # the parent CMD process).
  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"

  # Check that we have the expected version and architecture for Python
  - "python --version"
  - "python -c \"import struct; print(struct.calcsize('P') * 8)\""

  # Upgrade to the latest version of pip to avoid it displaying warnings
  # about it being out of date.
  - "python -m pip install --upgrade pip"
  - "pip install pipenv --upgrade-strategy=only-if-needed"
  - "pipenv --python=3.6 install --dev ."
  - "pipenv run pip list"

build: false

test_script:
  -  "pipenv run python -m pytest -v ./tests"
  -  "pipenv run black . --check"

artifacts:
  - path: dist\*

skip_tags: true

[hoefling]

Hi @j9ac9k, looks like you're also missing colorama installed from env marker. Try adding

- "pipenv run pip install colorama"

and see if it heals the test run.

[j9ac9k]

@hoefling looks like that was it!

In case you're curious here is the resulting build:

https://ci.appveyor.com/project/j9ac9k/timeview/build/1.0.49

In case you were wondering why I'm running the tests twice in two different ways, previously piping run python setup.py test would run but piping run python -m pytest -v tests would not.

[nicoddemus]

Thanks @hoefling for the summary and followup!

Since this is not related to pytest, I'm closing this issue. Thanks again.