Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate and address OOM for Windows + test-nlu-predictors on CI with TF 2.6 #9734

Closed
1 task
koernerfelicia opened this issue Sep 28, 2021 · 1 comment
Closed
1 task

Investigate and address OOM for Windows + test-nlu-predictors on CI with TF 2.6 #9734

koernerfelicia opened this issue Sep 28, 2021 · 1 comment
Assignees
@markus-hinsche

Comments

@koernerfelicia
Copy link
Contributor

koernerfelicia commented Sep 28, 2021 •
edited by rasabot-exalate
Loading

When we upgraded to TF 2.6, the test-nlu-predictors and test-other-unit-tests tests on Windows CI failed due to OOM. Since the upgrade is urgent, the temporary workaround was to lower JOBS to 1 for these tests (as opposed to 2), bringing the memory consumption below the limit of the runners.

We should investigate how to reduce the memory consumption of these tests, so that the tests for this configuration can be run in parallel again.

Please reach out to someone from Infra for support in using DataDog to investigate these.

Definition of Done
This was referenced Sep 28, 2021
Closed
Closed
@koernerfelicia koernerfelicia mentioned this issue Oct 25, 2021
Merged
2 tasks
@markus-hinsche markus-hinsche self-assigned this Feb 22, 2022
@markus-hinsche markus-hinsche mentioned this issue Mar 2, 2022
Closed
4 tasks
This was referenced Mar 16, 2022
Merged
Merged
@markus-hinsche
Copy link
Contributor

The following PRs will tackle this issue:

markus-hinsche added a commit that referenced this issue Mar 23, 2022
@markus-hinsche
Skip tests causing ResourceExhaustedError on Windows (#10992)
96b1d6b 
**Changes:**
- set JOBS=2 enabling running tests in parallel on Windows (Tackles: #9734)
- skip heavy test on windows because of `ResourceExhaustedError`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@markus-hinsche markus-hinsche

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@markus-hinsche @koernerfelicia