Running local tests: is nox the best way?

[rcomer]

📰 Custom Issue

If you follow the developer's installation instructions, it tells you to check your environment by running python setup.py test, with a link to the main testing page. But the main testing page tells you that setup.py test is deprecated and to use nox instead. nox makes its own environments though, so it isn't going to help you test the environment you just installed.

When the developers' guide was revamped last year, I really liked that it essentially said "here is a yml file that lets you easily make an environment that contains everything you need for testing". The guidance to use nox seems a bit orthogonal to that approach.

If I've understood, nox is something of a black box that will only run the pre-determined sets of tests. If I do get a test failure, I'd like to investigate it by e.g. poking at objects in an interactive python session, or re-running individual tests. It's not clear to me if and how I can do that with nox's environments.

I did have my first go at running the tests under nox yesterday. They seemed to run OK, but I got these errors after each session:

nox errors

Traceback (most recent call last):
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/managers.py", line 597, in _run_server
    server.serve_forever()
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/managers.py", line 173, in serve_forever
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/multiprocessing/util.py", line 133, in _remove_temp_dir
    rmtree(tempdir)
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "[git-path]/iris/.nox/tests-3-7/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfsc00002070b44262400000291'

All of this makes me think we should still be advertising a non-deprecated way to run the tests in a user-controlled environment. #3835 listed a few ways:

• python -m iris.tests.runner --default-tests
• python -m unittest discover iris

Then there's also pytest (though I just tried pointing pytest at the iris tests in a clean branch and got 6 failures/errors 😢 ). Obviously experienced devs may have their own preferred way to do it, but I think we should advertise something straightforward that newbie devs can just run.

[trexfeathers]

I don't disagree; it's important to be able to run subsets of tests on an ad-hoc basis, and I expect I've been personally spoiled by PyCharm so I'm ignorant of many problems in this area.

But I'd also like to note that I love that Nox gives a common entry point for developers and the CI - you can run locally exactly what the CI runs, in theory reducing nasty surprises when pushing PR's.

[rcomer]

I love that Nox gives a common entry point for developers and the CI

Oh yeah, I'm not saying we should remove the nox guidance. But at the moment the guide reads to me like nox is the only recommended way to do it.

[rcomer]

though I just tried pointing pytest at the iris tests in a clean branch and got 6 failures/errors 😢

#4192 proposes fixes for this.

[rcomer]

From discussion at #3835 it seems that pytest is the only game in town if you want to parallelise. Could we

• Fix the tests so they run consistently under pytest and nose/setuptools (Test wrangling vs v3.0.x #4249)
• Advertise pytest in the developers' guide as the way to run tests in a local user-controlled environment (in addition to nox guidance) Recommend pytest in developer guide #4866
• Add pytest and pytest-xdist to the requirements to maintain the "this yml gives you everything you need for testing" approach Replace nose with pytest #4734

?

[rcomer]

I'd be happy to PR something for the dev guide as I've thought about it previously. I'd also be happy to review if @bsherratt wants to put something up.

Would we be happy to add pytest and pytest-xdist to the requirements? The installation guide currently states

The requirements/ci/iris.yml file defines the Iris development conda environment name and all the relevant top level conda-forge package dependencies that you need to code, test, and build the documentation.

which makes me think they should be added if we're going to advertise using them for testing. I'm also aware though that there is a worry about "dependency bloat". I think we eventually want to use pytest in the CI (#3835), so perhaps these packages need to eventually go in the ymls anyway and I am overthinking it...?

[trexfeathers]

Would we be happy to add pytest and pytest-xdist to the requirements?

While we know pytest works (thanks to you!), we haven't explicitly adopted it yet. It's still possible to run all/some of the tests without pytest, so it's not an actual requirement. Later, yes. Right now, I wouldn't (IMHO).

[vsherratt]

Yeah I could have a think about some wording, though no guarantees that it would be any time soon.

Immediate thoughts:

• Most of the existing setuptools section can be easily adapted to refer to pytest instead
  • Make it clear that an additional conda install pytest step is needed after "installing from source"
  • Run all tests with pytest or specific tests with pytest [files]. Works regardless of working directory
• Do still mention setuptools, until pytest is "properly" adopted - which I agree is not yet, at the very least because I have no idea whether the additional options like -d have a pytest equivalent. But lose the example output, just a deprecation warning and basic how-to.
• Brief introduction under the "running the tests" header with links to pytest/nox(/setuptools) subheadings. Perhaps including why you'd use one or the other.

[rcomer]

I generally point pytest at lib/iris/tests, to be equivalent to what setuptools runs. If you just do pytest it also gets the gallery tests. That's not necessarily a bad thing, but there is advice elsewhere for running those.

[rcomer]

pytest and pytest-xdist are now in the requirements - #4734 🥳

[rcomer]

Dev guide now recommends pytest rather than python setup.py test. So I'll archive this.