How to run tox and poetry together against multiple dependency versions

[hugorodgerbrown]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Issue

I have a question around the use of poetry and tox, when testing a library against a matrix of supported dependencies. My current example is a Django app, which I would like to test against Django 2.2. and 3.0. The tox.ini config is as below (taken from the docs - https://python-poetry.org/docs/faq/#is-tox-supported):

[tox]
envlist = py{36,37,38}-django{22,30}

[testenv]
deps =
    django22: Django==2.2
    django30: Django==3.0

whitelist_externals = poetry
skip_install = true

commands =
    poetry install -vvv
    poetry run django-admin --version
    poetry run pytest tests/

My pyproject.toml file lists the Django dependency as:

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"

When running tox, the version number output for all test runs is 3.0. It looks like the lock file is created on the first test run, and then re-used for the rest, and that the lock file is always the latest version of Django (3.0), even when 2.2 is already installed.

I'm a bit stuck at this point, as without being able to test against a matrix of versions I can't progress. Any help gratefully received. (I am also happy to add a docs PR once I've worked out the solution, as I can't be the only person with this issue?)

---

Update: adding output from a test run that shows a.) Django 2.2 being installed by tox, and then b.) poetry overwriting it.

my-app$ tox -r -e py36-django22
py36-django22 recreate: .tox/py36-django22
py36-django22 installdeps: Django==2.2
py36-django22 installed: Django==2.2,my-app==0.1.0,pytz==2019.3,sqlparse==0.3.0
py36-django22 run-test: commands[0] | poetry install -vvv
Using virtualenv: .tox/py36-django22
Updating dependencies
Resolving dependencies...
   1: derived: django (^2.2 || ^3.0)
   ...
PyPI: 10 packages found for django >=2.2,<4.0
   ...
   1: Version solving took 3.330 seconds.
   1: Tried 1 solutions.

Writing lock file

Package operations: 52 installs, 1 update, 0 removals, 3 skipped

  - ...
  - Updating django (2.2 -> 3.0)
  - ...

[hugorodgerbrown]

Cross-posted to StackOverflow - https://stackoverflow.com/q/59377071/45698

[jeromebon]

Same issue, I'm to test against django 1.11 up to 3.0. But poetry always override the specified deps. One way to solve this would be to allow specifying a version to poetry install as long as it doesn't collide with the poetry.toml
In tox we could then use install_command = poetry install {packages}

[sinoroc]

I made a suggestion to hopefully solve this kind of issues on the StackOverflow question as well as on the issue #1941.

Since tox already takes care of installing the project and its dependencies in virtual environments, there is no need to get poetry involved. Except to build the sdist, which is exactly what isolated_build does.
Now the next issue is that tox doesn't know about poetry's dev-dependencies. A pattern I have seen commonly used to remedy this in other contexts, is to list such dependencies as an extra.

So it could look like the following, note how the extra test replaces the dev-dependencies and how the commands do not mention poetry at all:

tox.ini

[tox]
envlist = py{36,37,38}-django{22,30}
isolated_build = True

[testenv]
# ...
deps =
    django22: Django==2.2
    django30: Django==3.0
extras =
    test
commands =
    pytest

pyproject.toml

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = "^3.6"
django = "^2.2 || ^3.0"
#
pytest = { version = "^5.2", optional = true }

[tool.poetry.extras]
test = ["pytest"]

[build-system]
# ...

On the other hand, the drawback is that to get the development dependencies in the normal poetry virtual environment an extra call to poetry install --extras 'test' is required.

[wesleykendall]

I got around this issue by exporting all dependencies, removing Django, and then using pip to install them:

[tox]
isolated_build = true
envlist = py{36,37,38}-django{22,30}

[testenv]
deps =
    django22: Django>=2.2,<2.3
    django30: Django>=3.0,<3.1
whitelist_externals =
  poetry
  bash
skip_install = true
commands =
    bash -c 'poetry export --dev --without-hashes -f requirements.txt | grep -v "^[dD]jango==" > .requirements.txt'
    poetry run pip install --no-deps -r .requirements.txt
    poetry run django-admin --version
    poetry run pytest dir/

There may be a nicer way to run commands with pipe in tox.ini other than bash -c, but this works for me for just running tests. No changes to my pyproject.toml either.

[sinoroc]

Would you mind expanding on the advantages of your solution?

Anyway... Unless I am missing something, I believe this could probably be simplified:

• isolated_build shouldn't be necessary, since the project is not buil by tox, because of skip_install;
• poetry run shouldn't be necessary, since tox already does virtual environment isolation and pip installs the dependencies.

Maybe something like that:

[tox]
envlist = py{36,37,38}-django{22,30}

[testenv]
deps =
    django22: Django>=2.2,<2.3
    django30: Django>=3.0,<3.1
whitelist_externals =
  bash
skip_install = true
commands_pre =
    bash -c \'poetry export --dev --without-hashes -f requirements.txt | grep -v "^[dD]jango==" > .requirements.txt\'
    python -m pip install --no-deps -r .requirements.txt
    python -m pip install .
    python -m django --version
commands =
    python -m pytest dir/

Probably the bash part can be simplified as well.

[wesleykendall]

@sinoroc the main advantage is no need to change pyproject.toml. This is a work-around for those that don't want to maintain an "extras" part of pyproject.toml just for testing requirements, not to mention having to do poetry install --extras 'test' as mentioned previously. Of course one needs to keep in mind that pip install is not going to do conflict checking or anything else like poetry does. For my situation this doesn't matter since I have other checks in my continuous integration for this.

And yea, there are likely some ways to make the original code nicer. I just posted what was working for me after hacking around at it. You can also condense it more if you want to avoid storing a ".requirements.txt" file:

bash -c 'poetry export --dev --without-hashes -f requirements.txt | grep -v "^[dD]jango==" | poetry run pip install --no-deps -r /dev/stdin'
poetry run pytest ...

I leave poetry run prefixes in my examples because I have other parts of my continuous integration process after tox that use poetry for venv management. But yea you don't have to use poetry and can use normal pip

[sinoroc]

I see, but at this point, we are skipping so many of the advantages of tox, it might be worth questioning why use it at all. Oh by the way, you might want to add skipsdist = true as well.

But sure, no discussion about it, each project has its own specific needs that call for its own specific workflow. It's good we have the flexibility to adapt to each needs.

To me, the key element that pops out of this discussion, is that there is no straightforward way for tox to get the list of dev dependencies from poetry. And that is what is preventing a clean integration between these two tools.

[wesleykendall]

@sinoroc my suggestion is for people that don't want to use the previous suggestions. It doesn't skip many advantages of tox and is not that different than the original solutions. One could also say that the problem is also that poetry does not allow one to override dependencies in a poetry install, which is effectively what I'm doing by dynamically ignoring the Django poetry tries to install.

There's been quite a bit of noise since my previous suggestion. So, if you are coming to this thread and want a way to use tox in a poetry project without hacking your pyproject.toml, you can do this:

[tox]
isolated_build = true
envlist = py{36,37,38}-django{22,30}

[testenv]
deps =
    django22: Django>=2.2,<2.3
    django30: Django>=3.0,<3.1
whitelist_externals =
  poetry
  bash
skip_install = true
commands =
    bash -c 'poetry export --dev --without-hashes -f requirements.txt | grep -v "^[dD]jango==" | poetry run pip install --no-deps -r /dev/stdin'
    # An example command to show that it is the right Django version
    poetry run django-admin --version
    poetry run pytest dir/

[webartifex]

@wesleykendall
I am in a similar situation. Question: what happens if someone clones my project with your solution but does not have poetry installed? IMO that should break then, right?

So, if I want my pyproject.toml and tox.ini as generic as possible, there is no way to avoid specifying the dev dependencies twice?

Proposed way forward: Ask the tox dev's to include an option such that tox also installs the pyproject.toml dev dependencies?

[jlumbroso]

@wesleykendall Your solution is the most elegant one I've seen so far IMHO.

Here's how I would strive for tighter uncoupling right now:

• Have the dependency export happen on commit. I.e., on commit, export dependencies from Poetry as done here with GitHub Actions. This generates both files requirements.txt and requirements-dev.txt.
• Instead of:

  bash -c 'poetry export --dev --without-hashes -f requirements.txt | grep -v "^[dD]jango==" | poetry run pip install --no-deps -r /dev/stdin'

  one can then to:

  bash -c 'cat requirements-dev.txt | grep -v "^[dD]jango==" | pip install --no-deps -r /dev/stdin'

@webartifex The above tweak should address your issue since it no longer requires poetry when cloning/testing the package.

Other issues re: tox/poetry: #848, #1745, #1941.

[sinoroc]

I published on PyPI a plugin for Tox to instruct it to install Poetry's development dependencies in the test environments: tox-poetry-dev-dependencies. It's just a proof of concept. I didn't test much, only with a very simple project.

[johnthagen]

Edit: I've been using tox-poetry-installer and as of 0.5.0 it has been working very well for our team.

It would be helpful if there was some official docs at https://python-poetry.org/docs/faq/#is-tox-supported had some discussion about the issues around pyproject.toml dev-dependencies and tox.

So far, the best solution I've found that doesn't break tox's isolation by whitelisting poetry/bash/etc is to use @sinoroc's tox-poetry-dev-dependencies plugin. This allows using tox in a normal, isolated way without having to modify pyproject.toml. There are a few open issues (sinoroc/tox-poetry-dev-dependencies#48), but as poetry-core progresses, these should be solvable.

[sinoroc]

I think a good course of action, would be for people here (i.e. people who have experience with tox and poetry), to make some clear condensed suggestion of what they would like to see in the documentation/FAQ.

• keep it short
• keep it non opinionated
• keep it general enough that anyone can use it without being surprising / counter-intuitive
• keep it simple for beginners
• make it meaningful so that it is clear what task the poetry+tox combination achieves (test against multiple Python versions and/or test against multiple versions of a dependency)
• avoid suggesting 3rd party tool (I do not think tox plugins should be in the FAQ)

Someone already made a pull request: #2416

Maybe people should go have a look at it, comment on it, make suggestions, etc.

[jlumbroso]

Belatedly, thank you @sinoroc, your package takes care of this need very nicely! 🙏🏻