Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[wasm][aot] Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests fail while linking due to OOM #66647

Closed
ilonatommy opened this issue Mar 15, 2022 · 6 comments · Fixed by #67017
Closed

[wasm][aot] Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests fail while linking due to OOM #66647

ilonatommy opened this issue Mar 15, 2022 · 6 comments · Fixed by #67017
Labels
arch-wasm WebAssembly architecture area-Build-mono

Comments

@ilonatommy
Copy link
Member

ilonatommy commented Mar 15, 2022 •
edited
Loading

Run: run-extra-platforms Linux Release LibraryTests_AOT 20220312.4

Error:

emcc : error : '/datadisks/disk1/work/B43F097B/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -O2 --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/B43F097B/p/build/wasm/WasmApp.Native.targets(410,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]

Full log.

Last passing run: 20220310.1.
@dotnet-issue-labeler dotnet-issue-labeler bot added area-Infrastructure-mono untriaged New issue has not been triaged by the area owner labels Mar 15, 2022
@ghost
Copy link

ghost commented Mar 15, 2022

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: run-extra-platforms Linux Release LibraryTests_AOT 20220312.4

Error:

emcc : error : '/datadisks/disk1/work/B43F097B/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -O2 --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/B43F097B/p/build/wasm/WasmApp.Native.targets(410,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]

Last passing run: 20220310.1.

Author: ilonatommy
Assignees: -
Labels:

untriaged, area-Infrastructure-mono
Milestone: -

@ilonatommy
Copy link
Member Author

ilonatommy commented Mar 15, 2022 •
edited
Loading

@ViktorHofer, could you check if 0046648 update could not cause it?

@ShreyasJejurkar
Copy link
Contributor

Tagging correctly, @ViktorHofer :)

@ViktorHofer
Copy link
Member

@ViktorHofer, could you check if 0046648 update could not cause it?

0046648 is most definitely unrelated as msquic is networking related and was the only component touched in the change that you linked to.

@radekdoulik
Copy link
Member

Looks like wasm-opt is getting out of memory.

wasm-opt [ 7359.203268] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/helix.service,task=wasm-opt,pid=28704,uid=1000 [ 7359.203288] Out of memory: Killed process 28704 (wasm-opt) total-vm:6024128kB, anon-rss:5859504kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:11612kB oom_score_adj:0 [ 7359.406901] oom_reaper: reaped process 28704 (wasm-opt), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB```

@ilonatommy ilonatommy changed the title AOT build failure - Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests AOT build failure - out of memory Mar 15, 2022
@radical radical changed the title AOT build failure - out of memory [wasm][aot] Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests fail while linking due to OOM Mar 16, 2022
@radical radical added arch-wasm WebAssembly architecture area-Build-mono labels Mar 16, 2022
@ghost
Copy link

ghost commented Mar 16, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: run-extra-platforms Linux Release LibraryTests_AOT 20220312.4

Error:

emcc : error : '/datadisks/disk1/work/B43F097B/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -O2 --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/B43F097B/p/build/wasm/WasmApp.Native.targets(410,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/B43F097B/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/B43F097B/w/A1780928/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/B43F097B/w/A1780928/e/publish/ProxyProjectForAOTOnHelix.proj]

Full log.

Last passing run: 20220310.1.

Author: ilonatommy
Assignees: -
Labels:

arch-wasm, untriaged, area-Build-mono, area-Infrastructure-mono
Milestone: -

@radical radical removed area-Infrastructure-mono untriaged New issue has not been triaged by the area owner labels Mar 16, 2022
@radical radical mentioned this issue Mar 23, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Mar 23, 2022
radical added a commit that referenced this issue Mar 25, 2022
@radical
[wasm][aot] Run the high resource AOT tests on build machine itself (#…
286492c 
…67017)

Some library tests fail to AOT on helix by getting oomkill'ed. Since the
build machine have more resources than helix, we can AOT these specific
library tests there, and send the built output to helix to run.

Specifically:
- `System.Text.Json.SourceGeneration.Roslyn3.11.Tests`
- `System.Text.Json.SourceGeneration.Roslyn4.0.Tests`
- `System.Text.Json.Tests`
- `Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests` 

This adds two new jobs to `runtime-wasm`, and for rolling builds - `LibraryTests_HighResource_AOT` for linux, and windows.

Fixes #65356 
Fixes #65411
Fixes #61524 
Fixes #66647
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Mar 25, 2022
radekdoulik pushed a commit to radekdoulik/runtime that referenced this issue Mar 30, 2022
@radical @radekdoulik
[wasm][aot] Run the high resource AOT tests on build machine itself (d…
96fdef6 
…otnet#67017)

Some library tests fail to AOT on helix by getting oomkill'ed. Since the
build machine have more resources than helix, we can AOT these specific
library tests there, and send the built output to helix to run.

Specifically:
- `System.Text.Json.SourceGeneration.Roslyn3.11.Tests`
- `System.Text.Json.SourceGeneration.Roslyn4.0.Tests`
- `System.Text.Json.Tests`
- `Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests` 

This adds two new jobs to `runtime-wasm`, and for rolling builds - `LibraryTests_HighResource_AOT` for linux, and windows.

Fixes dotnet#65356 
Fixes dotnet#65411
Fixes dotnet#61524 
Fixes dotnet#66647
@ghost ghost locked as resolved and limited conversation to collaborators Apr 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-wasm WebAssembly architecture area-Build-mono
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[wasm][aot] Run the high resource AOT tests on build machine itself radical/runtime
5 participants
@radical @radekdoulik @ViktorHofer @ShreyasJejurkar @ilonatommy