Speed up tests on CircleCI and use pytest to run them #1804

bouweandela · 2020-09-02T14:06:47Z

This pull request contains the following changes

Replace the deprecated python setup.py test command with pytest
Use pytest-xdist to run tests in parallel
Use Codacy orb on CircleCI for uploading test coverage

Tasks

Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
Give this pull request a descriptive title that can be used as a one line summary in a changelog
Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
Please use yamllint to check that your YAML files do not contain mistakes

Closes #1671

valeriupredoi · 2020-09-02T14:51:14Z

looking at it right now, boss 👍

valeriupredoi · 2020-09-02T14:55:47Z

not liking it on Jasmin:

(esmvaltool) [valeriu@jasmin-sci5 esmvaltool]$ pytest
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --numprocesses
  inifile: /home/users/valeriu/esmvaltool/setup.cfg
  rootdir: /home/users/valeriu/esmvaltool

pytest=5.4.1. Investigating now...

valeriupredoi · 2020-09-02T15:21:55Z

OK proves out I didn't have pytest-xdist but after having installed it I get this beauty:

platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/users/valeriu/esmvaltool, configfile: setup.cfg
plugins: cov-2.8.1, env-0.6.2, flake8-1.0.4, metadata-1.8.0, html-2.1.1, mock-3.1.1, forked-1.3.0, xdist-2.1.0
gw0 C / gw1 IINTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/_pytest/main.py", line 238, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/xdist/dsession.py", line 78, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/xdist/workermanage.py", line 65, in setup_nodes
INTERNALERROR>     return [self.setup_node(spec, putevent) for spec in self.specs]
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/xdist/workermanage.py", line 65, in <listcomp>
INTERNALERROR>     return [self.setup_node(spec, putevent) for spec in self.specs]
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/xdist/workermanage.py", line 73, in setup_node
INTERNALERROR>     node.setup()
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/xdist/workermanage.py", line 260, in setup
INTERNALERROR>     self.config.hook.pytest_configure_node(node=self)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pytest_cov/plugin.py", line 222, in pytest_configure_node
INTERNALERROR>     self.cov_controller.configure_node(node)
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pytest_cov/engine.py", line 236, in configure_node
INTERNALERROR>     workerinput(node).update({
INTERNALERROR>   File "/home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pytest_cov/compat.py", line 42, in fn
INTERNALERROR>     return getattr(obj, attr, *args)
INTERNALERROR> AttributeError: 'WorkerController' object has no attribute 'slaveinput'

BTW @bouweandela you need to tell people they have to eithet pip install -e .[develop] or install whatevs the testing seesion needs before they run the tests - that's why I like python setup.py test better, dynamical installation is nice

valeriupredoi · 2020-09-02T15:36:34Z

ah that's a pytest-cov issue pytest-dev/pytest-xdist#577

valeriupredoi · 2020-09-02T15:52:48Z

OK it proves out you need pytest-cov>2.8.1 since a bug only got fixed later. But bummer again:

(esmvaltool) [valeriu@jasmin-sci5 esmvaltool]$ pytest
========================================================== test session starts ==========================================================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/users/valeriu/esmvaltool, configfile: setup.cfg
plugins: env-0.6.2, flake8-1.0.4, metadata-1.8.0, html-2.1.1, mock-3.1.1, forked-1.3.0, xdist-2.1.0, cov-2.10.1
gw0 [0] / gw1 [0]s / 1 error                                                                                                            
Coverage.py warning: No data was collected. (no-data-collected)
Coverage.py warning: No data was collected. (no-data-collected)

================================================================ ERRORS =================================================================
_____________________________________________________ ERROR collecting test session _____________________________________________________
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pytest_flake8.py:65: in pytest_collect_file
    return Flake8Item(
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/_pytest/nodes.py:95: in __call__
    warnings.warn(NODE_USE_FROM_PARENT.format(name=self.__name__), stacklevel=2)
E   pytest.PytestDeprecationWarning: Direct construction of Flake8Item has been deprecated, please use Flake8Item.from_parent.
E   See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent for more details.
_____________________________________________________ ERROR collecting test session _____________________________________________________
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/pytest_flake8.py:65: in pytest_collect_file
    return Flake8Item(
../anaconda3R/envs/esmvaltool/lib/python3.8/site-packages/_pytest/nodes.py:95: in __call__
    warnings.warn(NODE_USE_FROM_PARENT.format(name=self.__name__), stacklevel=2)
E   pytest.PytestDeprecationWarning: Direct construction of Flake8Item has been deprecated, please use Flake8Item.from_parent.
E   See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent for more details.
-------------------------- generated html file: file:///home/users/valeriu/esmvaltool/test-reports/report.html --------------------------

----------- coverage: platform linux, python 3.8.2-final-0 -----------
Coverage HTML written to dir test-reports/coverage_html
Coverage XML written to file test-reports/coverage.xml

======================================================== short test summary info ========================================================
ERROR  - pytest.PytestDeprecationWarning: Direct construction of Flake8Item has been deprecated, please use Flake8Item.from_parent.
ERROR  - pytest.PytestDeprecationWarning: Direct construction of Flake8Item has been deprecated, please use Flake8Item.from_parent.
========================================================== 2 errors in 23.44s ======

bouweandela · 2020-09-02T15:53:35Z

Could you try with an up to date environment, maybe just run pip install -e '.[develop]' again before running the pytest command?

you need to tell people they have to eithet pip install -e .[develop] or install whatevs the testing seesion needs before they run the tests

That's why the installation instructions come before how to run the tests (or how to use any of the other tools we recommend for developing) in the contribution guidelines.

that's why I like python setup.py test better, dynamical installation is nice

I also liked it, but it's deprecrated pypa/setuptools#1684

valeriupredoi · 2020-09-02T15:55:48Z

yeah I did, I have the latest env after conda env updating and pip install -e .[develop] -ing, man. It's a number of issues that are independent of your implementation, see e.g. the pytest-cov one. Note that after upading the env pytest-cov still stayed at 2.8.1

valeriupredoi · 2020-09-02T16:01:30Z

gah, updating flake8 to 3.8.3 is still not resolving that last issue, last resort - recreating the environment from scratch. Note however that you'll need to pin pytest-cov>2.8.1 that's for sure for now

valeriupredoi · 2020-09-02T16:55:04Z

OK good news! I recreated the env from scratch and tests run fine now, here's the difference in the envs (the older env esmvaltool at the stage of the last Flake8 error I posted above, esmvaltool-tests is the new env I just built):

(esmvaltool-tests) [valeriu@jasmin-sci5 esmvaltool]$ conda list pytest
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvaltool-tests:
#
# Name                    Version                   Build  Channel
pytest                    6.0.1                    pypi_0    pypi
pytest-cov                2.10.1                   pypi_0    pypi
pytest-env                0.6.2                    pypi_0    pypi
pytest-flake8             1.0.6                    pypi_0    pypi
pytest-forked             1.3.0                    pypi_0    pypi
pytest-html               2.1.1                    pypi_0    pypi
pytest-metadata           1.10.0                   pypi_0    pypi
pytest-xdist              2.1.0                    pypi_0    pypi
(esmvaltool-tests) [valeriu@jasmin-sci5 esmvaltool]$ conda list flake8
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvaltool-tests:
#
# Name                    Version                   Build  Channel
flake8                    3.8.3                    pypi_0    pypi
flake8-polyfill           1.0.2                    pypi_0    pypi
pytest-flake8             1.0.6                    pypi_0    pypi
(esmvaltool-tests) [valeriu@jasmin-sci5 esmvaltool]$ conda deactivate
(base) [valeriu@jasmin-sci5 esmvaltool]$ conda activate esmvaltool
(esmvaltool) [valeriu@jasmin-sci5 esmvaltool]$ conda list pytest
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
pytest                    6.0.1                    pypi_0    pypi
pytest-cov                2.10.1                   pypi_0    pypi
pytest-env                0.6.2                    pypi_0    pypi
pytest-flake8             1.0.4                    pypi_0    pypi
pytest-forked             1.3.0                    pypi_0    pypi
pytest-html               2.1.1                    pypi_0    pypi
pytest-metadata           1.8.0                    pypi_0    pypi
pytest-mock               3.1.1                    pypi_0    pypi
pytest-xdist              2.1.0                    pypi_0    pypi
(esmvaltool) [valeriu@jasmin-sci5 esmvaltool]$ conda list flake8
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
flake8                    3.8.3                    pypi_0    pypi
pytest-flake8             1.0.4                    pypi_0    pypi

so I think we need to pin pytest-flake8>1.0.4 too. Just a heads up this is with the new Python 3.8.5, as opposed to esmvaltool's Python 3.8.2, so if the Python version is giving us headaches, then we're hosed 🐍

valeriupredoi · 2020-09-03T13:02:27Z

I can't reproduce CicleCI's fail at conda build stage, I ran it fine on my laptop - seems to be an issue with Julia that is creeping up only on the CI machine, weird

valeriupredoi · 2020-09-03T13:36:05Z

what I can do is rerun the test several times over on the CI machine and I notice it dies at different stages during the R packages installation but always after 16min - is there a timeout on the CI machine for a test?

valeriupredoi · 2020-09-03T13:37:29Z

ah! there we go https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-hit-timeout-limit - and the stoopid R install output doesnt flush to stdout only at the very end

valeriupredoi · 2020-09-03T13:41:12Z

gah! you've actually placed the condition in the config...why th is it dying then? Superconfused 😕

valeriupredoi · 2020-09-03T13:56:05Z

son of a gun it's the memory https://support.circleci.com/hc/en-us/articles/115014359648-Exit-code-137-Out-of-memory - any clue how to go about this?

valeriupredoi · 2020-09-03T14:06:22Z

also @bouweandela can you pls not name the branch with the same name as a dir in ESMValTool - git was going bazooka trying to checkout tests and taking me instantly back onto master 😁

bouweandela · 2020-09-03T15:59:20Z

son of a gun it's the memory .. any clue how to go about this?

There's a few options:

Pay
Make a pull request and fix the conda dependency solver
Accept that the tests fail occasionally

For now I went for the latter option ;-)

bouweandela · 2020-09-03T16:00:47Z

You could run the conda build on your laptop if you want to check if it works

valeriupredoi · 2020-09-03T16:24:29Z

You could run the conda build on your laptop if you want to check if it works

I did and it works fine. Maybe I should stop posting comments like it's Whatsapp here 😁 BTW maybe we should start thinking about upgrading to a paid Circle account? Not off my credit card though, but surely there's a way - mooch off ISENES maybe? 😆

valeriupredoi · 2020-09-03T16:31:24Z

I'll plug in the Github Actions tests for ESMValTool starting tomorrow so maybe this will be tested proper and w/o $$$ there, let's see

valeriupredoi · 2020-09-04T11:07:59Z

@bouweandela you wanna merge this? I am piggybacking on it in #1806 and will be nice to have this in master before I finalize the GA one 🍺

bouweandela · 2020-09-04T12:43:48Z

Thanks for reviewing!

valeriupredoi · 2020-09-04T13:16:38Z

good stuff, B-dude! 🍺

bouweandela added 3 commits September 2, 2020 15:17

Use pytest-xdist, avoid setup.py test and update CircleCI config

0572f90

Try to use codacy coverage orb

739f439

Remove test alias

2ecabd6

bouweandela requested a review from valeriupredoi September 2, 2020 14:06

Require minimal version of pytest-cov

400def4

Require minimum version of pytest-flake8

da25e63

valeriupredoi mentioned this pull request Sep 3, 2020

Run tests in parallel and when building conda package ESMValGroup/ESMValCore#745

Merged

4 tasks

extra mem req

84c1bfc

removed mem req coz they are not for free accounts blehhh

41908f9

valeriupredoi approved these changes Sep 4, 2020

View reviewed changes

bouweandela merged commit 3117b20 into master Sep 4, 2020

bouweandela deleted the tests branch September 4, 2020 12:43

bouweandela added the deprecated feature label Jan 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up tests on CircleCI and use pytest to run them #1804

Speed up tests on CircleCI and use pytest to run them #1804

bouweandela commented Sep 2, 2020 •

edited

Loading

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

bouweandela commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

bouweandela commented Sep 3, 2020

bouweandela commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 4, 2020

bouweandela commented Sep 4, 2020

valeriupredoi commented Sep 4, 2020

Speed up tests on CircleCI and use pytest to run them #1804

Speed up tests on CircleCI and use pytest to run them #1804

Conversation

bouweandela commented Sep 2, 2020 • edited Loading

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

bouweandela commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 2, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

bouweandela commented Sep 3, 2020

bouweandela commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 3, 2020

valeriupredoi commented Sep 4, 2020

bouweandela commented Sep 4, 2020

valeriupredoi commented Sep 4, 2020

bouweandela commented Sep 2, 2020 •

edited

Loading