First implementation of tox_to_nox for tox 4

[frenzymadness]

Fixes: #673

It seems to me that the easiest way forward is to call tox config, parse the output and then preprocess it into a dictionary so it can be easily used in the jinja template.

What do you think?

From this tox.ini:

[tox]
envlist = black,mypy,py{36,37,38,39,310,311,312}
isolated_build = True

[testenv]
commands =
    python -m platform
    pytest -v {posargs} test.py
deps =
    pytest

[testenv:black]
description="better than blue"
skip_install=true
deps=black
commands=black --check --diff --extend-exclude=".tox|test" ./

[testenv:mypy]
use_develop=true
deps=mypy
commands=mypy --strict -p marshalparser

it now generates this noxfile.py

import nox


@nox.session()
def DEFAULT(session):
    """"""
    session.install('.')

@nox.session(python='/home/lbalhar/.virtualenvs/nox/bin/python')
def testenv_black(session):
    """"better than blue""""
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('black')
    session.run('black --check --diff '--extend-exclude=.tox|test' ./')

@nox.session(python='/home/lbalhar/.virtualenvs/nox/bin/python')
def testenv_mypy(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('mypy')
    session.install('-e', '.')
    session.run('mypy --strict -p marshalparser')

@nox.session(python='py310')
def testenv_py310(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py311')
def testenv_py311(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py312')
def testenv_py312(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py36')
def testenv_py36(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py37')
def testenv_py37(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py38')
def testenv_py38(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

@nox.session(python='py39')
def testenv_py39(session):
    session.env['PIP_DISABLE_PIP_VERSION_CHECK'] = '1'
    session.env['PYTHONIOENCODING'] = 'utf-8'
    session.install('pytest')
    session.install('.')
    session.run('python -m platform')
    session.run('pytest -v test.py')

I guess that:

• the default env variables should be skipped.
• the default environment should be skipped.

Correct?

[FollowTheProcess]

Hey @frenzymadness thanks for the PR! I agree with the approach of parsing the config from the command line, I think that's realistically the best we can do for tox 4 compatibility.

Just a few comments from my side but this looks like a great start 👍🏻 Will need some tests before it's ready to merge but appreciate the draft to sign off the approach

[frenzymadness]

Right now, all the tests are happy with both tox versions. I skipped just the test of the invalid identified in the env name because it seems that tox 4 does not support it.

Another problem is how to include tests with tox 3 into nox's tests. I was thinking about parametrizing the main test session but it seems to be a waste of resources to run all the tests again with tox 3 if the vast majority doesn't depend on tox. My solution is to downgrade tox to version <4 and then run the pytest again in the same session but only the tests of tox_to_nox.

[FollowTheProcess]

Just looks like a couple of uncovered sections in tox_to_nox

[frenzymadness]

Just looks like a couple of uncovered sections in tox_to_nox

What do you mean? I'm looking at both jinja2 templates side-by-side and they both seem to handle the same configuration options from tox.ini.

[FollowTheProcess]

Sorry I meant in the tests running in CI, e.g.: https://github.com/wntrblm/nox/actions/runs/4004488540/jobs/6875460666 coverage is saying some things aren't covered 👍🏻

[frenzymadness]

The problem reported by coverage was actually caused by two conditions that were never false so I can remove them, nice. All linter should be happy now, at least they are happy on my machine.

[frenzymadness]

And now we have a different problem with dependencies. argcomplete 2.0.0 requires importlib-metadata<5,>=0.23; python_version == "3.7" but the latest tox depends on importlib-metadata>=6; python_version < "3.8" which means that they cannot be installed together in an environment with Python 3.7 and therefore both pytest runs tests with tox 3 and hence the coverage is less than 100 %.

We can force install tox 4 in a dedicated step in Python 3.7 env but there is a risk that it breaks nox or its dependencies.

I'd set a lower coverage limit for Python 3.7 or skip the coverage there entirely. What do you think?

[frenzymadness]

@FollowTheProcess what do you think about the proposed solutions?

[frenzymadness]

Force installation of Python 3.7 should fix the problem and it seems that it does not break anything at least for now.

[FollowTheProcess]

Hmm, so this would mean anyone installing Nox on 3.7 would never get tox 4 right? So if that's the case I don't think we need to be super strict on getting to 100% because users would never hit that branch either?

I'd personally be happy to mark that branch as excluded from coverage when on 3.7 (provided users installing Nox on 3.7 would never get tox 4 because of the dependency conflict above) with a nice explanatory comment etc. I think coverage allows you to do this, not 100% sure

[frenzymadness]

Let's see what will CI say now.

[frenzymadness]

The problem here is in the way how the Github Actions are configured. If you run nox test locally, it's green because the coverage report is combined from all runs with all Python versions and therefore the lower overage from Python 3.7 is not a problem. But in GH actions, the coverage report is divided into jobs for each tested Python version which makes the job for Python 3.7 fail on too low coverage.

I see two ways forward:

1. Do not run coverage for Python 3.7 on GH actions.
2. Reconfigure GH actions to collect coverage for individual jobs and run coverage report for the collected data in a separate job.

What do you prefer?

[frenzymadness]

I've implemented what I think is the simplest solution here - no session.notify("cover") for Python 3.7. That way, the coverage report is run only if called explicitly or if tests run for more than this one environment which should also fix the coverage so it should be green all the time.

[frenzymadness]

But now we are back at the problem with coverage in GH action. If we split the jobs in GH action in the same way we have them in the parametrized sessions in the noxfile, the coverage will fail for all of them because they won't be tested with both tox versions at the same time.

I'm going to enable GH actions in my fork and try to implement a single job for the coverage reporting.

[frenzymadness]

Finally, after many failures, it's done. Now, each session in noxfile has its own job in GH actions. The produced coverage files are uploaded by the individual jobs. When all testing jobs finish, the coverage job starts, downloads the files, combines them and provides the report.

Locally, it no longer makes sense to run coverage if you test only one environment, because it cannot be 100 %, so the notification of coverage session has been removed.

You can see the last run in my fork here: https://github.com/frenzymadness/nox/actions/runs/4224370541

[mtelka]

tox 3 is dead. Please go forward and merge this. Thank you.

[crwilcox]

@FollowTheProcess I think it might be worth 'resurrecting' this and merging. There are some straightforward conflicts to handle, and this would be an improvement for users. If you are also on board I can get this over the line if needed.

Thanks @frenzymadness for picking this up.

[crwilcox]

I'm not sure why this is needed, and if it would have a negative effect on old tox (tox <= 3)

[crwilcox]

Verified purpose.

[FollowTheProcess]

Yeah, sorry been swamped for a while and not got chance to be around as much as I'd have liked! I'm on board 👍🏻 I'll admit though I've never used tox so not super familiar with the syntax conversions

[crwilcox]

Verified purpose.