Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test_training.py speed on github actions CI #231

Closed
GenevieveBuckley opened this issue Oct 16, 2023 · 1 comment · Fixed by #233
Closed

Improve test_training.py speed on github actions CI #231

GenevieveBuckley opened this issue Oct 16, 2023 · 1 comment · Fixed by #233
Assignees

Comments

@GenevieveBuckley
Copy link
Collaborator

The github actions CI runs very slowly on the master branch because of the test_training.py file. We would like to improve the speed, and/or skip the unreasonably slow tests on github actions.

On the dev branch, without test_training.py the CI job finishes in about 6 minutes. But on the master branch including the test_training.py file, it takes over half an hour.

Additionally, I often saw the CI job killed with an error. The error did not describe why the job failed.

  • I don't think it was a timeout, by default github actions has a 6 hour timeout, and the lowest timeout I set was 1 hour.
  • Maybe the CI machine ran out of memory? This seems possible, and is something to check.

Comment from PR 223

Update: I think it's the test/test_training.py file that was making the CI so slow.

Yes, unfortunately test_training.py is quite slow.

A possible workaround would be to mark the super slow tests as pytest.mark.slow and skip them in the CI jobs.

Yes, that's one possibility. Before that I want to check if I can improve this test a bit to speed it up on CI and so that we can just run it with the normal tests.
(I developed the test on my notebook, and it was fast enough, but I guess the CPU in the CI is weaker, so that it takes quite long.)
...
I will then:
look into speeding up test_training.py (or mark it as pytest.mark.slow if I can't speed it up without making the test worse)

@constantinpape
Copy link
Contributor

I have stripped down test_training in #233 now. It still works as expected and everything runs much faster now (ca. 5 minutes for all tests and ca. 8 minutes for the complete CI).

I think that's acceptable, so I will close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants