Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] Optimize output from threaded test runner #1149

Merged
merged 13 commits into from
Feb 6, 2024

Conversation

maraf
Copy link
Member

@maraf maraf commented Feb 2, 2024

  • Pass verbosity from xharness server to the browser
  • Use logger instead of Console to respect verbosity
  • Drop custom message process from xharness server
  • Log test output as Debug
  • Align test ouput for threaded and threadless runners

Contributes to dotnet/runtime#88763

Threadless - ok
info: Application arguments: --run WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll -method System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success -notrait category=OuterLoop -notrait category=failing -verbosity Info
info: [main 17:01:43.503] MONO_WASM: starting script http://127.0.0.1:63389/_framework/dotnet.js
info: [main 17:01:43.504] MONO_WASM: starting in http://127.0.0.1:63389/_framework/
info: Initializing dotnet version 9.0.0-dev commit hash f1583e0558e353d8de0f2816c9d7aa1c89fbe999
info: Discovering: System.Net.WebSockets.Client.Tests.dll (method display = ClassAndMethod, method display options = None)
info: Discovered:  System.Net.WebSockets.Client.Tests.dll (found 1 of 247 test cases)
info: Starting:    System.Net.WebSockets.Client.Tests.dll
warn: Blocking on the main thread is very dangerous, see https://emscripten.org/docs/porting/pthreads.html#blocking-on-the-main-browser-thread
info: Finished:    System.Net.WebSockets.Client.Tests.dll
info: Received expected 1462 of C:\Development\dotnet\runtime\artifacts\bin\System.Net.WebSockets.Client.Tests\Debug\net9.0-browser\browser-wasm\AppBundle\xharness-output\testResults.xml
info: Finished writing 1462 bytes of RESULTXML
info: Xml file was written to the provided writer.
info: Tests run: 2 Passed: 2 Inconclusive: 0 Failed: 0 Ignored: 0 Skipped: 0
info: test-main.js exiting WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll with result 0
info: [0x000f3b0c 17:01:46.373] MONO_WASM: forceDisposeProxies done: 6 imports, 0 exports, 0 GCHandles, 0 JSHandles.
info: WASM EXIT 0
info: Waiting to flush log messages with a timeout of 120 secs ..
XHarness exit code: 0
Threadless - error
info: Incoming arguments: --setenv=DOTNET_TEST_WEBSOCKETHOST=127.0.0.1:59788 --setenv=DOTNET_TEST_HTTPHOST=127.0.0.1:59788 --setenv=DOTNET_TEST_REMOTE_LOOP_HOST=127.0.0.1:59788 --setenv=DOTNET_TEST_SECUREWEBSOCKETHOST=127.0.0.1:59789 --setenv=DOTNET_TEST_SECUREHTTPHOST=127.0.0
.1:59789 --setenv=DOTNET_TEST_HTTP2HOST=127.0.0.1:59789 --setenv=XHARNESS_LOG_TEST_START=true --no-memory-snapshot --no-memory-snapshot --setenv=IsBrowserThreadingSupported=true --run WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll -method System.Net.WebSockets.Clien
t.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success -notrait category=OuterLoop -notrait category=failing -verbosity Info
info: Application arguments: --run WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll -method System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success -notrait category=OuterLoop -notrait category=failing -verbosity Info
info: [main 12:39:18.296] MONO_WASM: starting script http://127.0.0.1:59788/_framework/dotnet.js
info: [main 12:39:18.297] MONO_WASM: starting in http://127.0.0.1:59788/_framework/
info: Initializing dotnet version 9.0.0-dev commit hash f1583e0558e353d8de0f2816c9d7aa1c89fbe999
info: Discovering: System.Net.WebSockets.Client.Tests.dll (method display = ClassAndMethod, method display options = None)
info: Discovered:  System.Net.WebSockets.Client.Tests.dll (found 1 of 247 test cases)
info: Starting:    System.Net.WebSockets.Client.Tests.dll
info: [STRT] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: ws://127.0.0.1:59788/WebSocket/EchoWebSocket.ashx)
info: [FAIL] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: ws://127.0.0.1:59788/WebSocket/EchoWebSocket.ashx)
info: Assert.True() Failure
info: Expected: True
info: Actual:   False
info:    at System.Net.WebSockets.Client.Tests.SendReceiveTest.SendReceive_Concurrent_Success(Uri server) in C:\Development\dotnet\runtime\src\libraries\System.Net.WebSockets.Client\tests\SendReceiveTest.cs:line 398
info: --- End of stack trace from previous location ---
info: [STRT] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: wss://127.0.0.1:59789/WebSocket/EchoWebSocket.ashx)
info: [FAIL] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: wss://127.0.0.1:59789/WebSocket/EchoWebSocket.ashx)
info: Assert.True() Failure
info: Expected: True
info: Actual:   False
info:    at System.Net.WebSockets.Client.Tests.SendReceiveTest.SendReceive_Concurrent_Success(Uri server) in C:\Development\dotnet\runtime\src\libraries\System.Net.WebSockets.Client\tests\SendReceiveTest.cs:line 398
info: --- End of stack trace from previous location ---
info: Finished:    System.Net.WebSockets.Client.Tests.dll
info: Received expected 2178 of C:\Development\dotnet\runtime\artifacts\bin\System.Net.WebSockets.Client.Tests\Debug\net9.0-browser\browser-wasm\AppBundle\xharness-output\testResults.xml
info: Finished writing 2178 bytes of RESULTXML
info: Xml file was written to the provided writer.
info: Tests run: 2 Passed: 0 Inconclusive: 0 Failed: 2 Ignored: 0 Skipped: 0
info: test-main.js exiting WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll with result 1
info: WASM EXIT 1
info: Waiting to flush log messages with a timeout of 120 secs ..
fail: Application has finished with exit code TESTS_FAILED but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)
Threaded - ok
info: Application arguments: --run WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll -method System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success -threads -notrait category=OuterLoop -notrait category=failing -verbosity Info
info: [main 17:00:23.642] MONO_WASM: starting script http://127.0.0.1:63324/_framework/dotnet.js
info: [main 17:00:23.642] MONO_WASM: starting in http://127.0.0.1:63324/_framework/
info: Initializing dotnet version 9.0.0-dev commit hash f1583e0558e353d8de0f2816c9d7aa1c89fbe999
info: Configured filters:
info:   XUnitFilter [Type: Single; include; Method: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success]
info:   XUnitFilter [Type: Trait; exclude; Trait name: category; Trait value: OuterLoop]
info:   XUnitFilter [Type: Trait; exclude; Trait name: category; Trait value: failing]
info:   XUnitFilter [Type: Single; include; Method: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success]
info: Assembly: System.Net.WebSockets.Client.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (System.Net.WebSockets.Client.Tests.dll)
warn: Blocking on the main thread is very dangerous, see https://emscripten.org/docs/porting/pthreads.html#blocking-on-the-main-browser-thread
info: [Test environment: 32-bit .NET .NET 9.0 [collection-per-class, non-parallel]]
info: [Test framework: xUnit.net 2.6.7.0]
info:
info: Test collection for System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest
info: [PASS] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success
info: [PASS] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success
info: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest 0.1845299 ms
info: Received expected 1462 of C:\Development\dotnet\runtime\artifacts\bin\System.Net.WebSockets.Client.Tests\Debug\net9.0-browser\browser-wasm\AppBundle\xharness-output\testResults.xml
info: Finished writing 1462 bytes of RESULTXML
info: Xml file was written to the provided writer.
info: Tests run: 248 Passed: 2 Inconclusive: 0 Failed: 0 Ignored: 246 Skipped: 0
info: test-main.js exiting WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll with result 0
info: [0x000f3b0c 17:00:26.707] MONO_WASM: forceDisposeProxies done: 6 imports, 0 exports, 0 GCHandles, 0 JSHandles.
info: WASM EXIT 0
info: Waiting to flush log messages with a timeout of 120 secs ..
XHarness exit code: 0
Threaded - error
info: Application arguments: --run WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll -method System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success -threads -notrait category=OuterLoop -notrait category=failing -verbosity Info
info: [main 13:43:40.505] MONO_WASM: starting script http://127.0.0.1:61321/_framework/dotnet.js
info: [main 13:43:40.505] MONO_WASM: starting in http://127.0.0.1:61321/_framework/
info: Initializing dotnet version 9.0.0-dev commit hash f1583e0558e353d8de0f2816c9d7aa1c89fbe999
info: Configured filters:
info:   XUnitFilter [Type: Single; include; Method: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success]
info:   XUnitFilter [Type: Trait; exclude; Trait name: category; Trait value: OuterLoop]
info:   XUnitFilter [Type: Trait; exclude; Trait name: category; Trait value: failing]
info:   XUnitFilter [Type: Single; include; Method: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success]
info: Assembly: System.Net.WebSockets.Client.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (System.Net.WebSockets.Client.Tests.dll)
warn: Blocking on the main thread is very dangerous, see https://emscripten.org/docs/porting/pthreads.html#blocking-on-the-main-browser-thread
info: [Test environment: 32-bit .NET .NET 9.0 [collection-per-class, non-parallel]]
info: [Test framework: xUnit.net 2.6.7.0]
info:
info: Test collection for System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest
info: [STRT] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: ws://127.0.0.1:61321/WebSocket/EchoWebSocket.ashx)
info: [FAIL] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: ws://127.0.0.1:61321/WebSocket/EchoWebSocket.ashx)
info: Assert.True() Failure
info: Expected: True
info: Actual:   False
info:    at System.Net.WebSockets.Client.Tests.SendReceiveTest.SendReceive_Concurrent_Success(Uri server) in C:\Development\dotnet\runtime\src\libraries\System.Net.WebSockets.Client\tests\SendReceiveTest.cs:line 418
info: --- End of stack trace from previous location ---
info: [STRT] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: wss://127.0.0.1:61322/WebSocket/EchoWebSocket.ashx)
info: [FAIL] System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.SendReceive_Concurrent_Success(server: wss://127.0.0.1:61322/WebSocket/EchoWebSocket.ashx)
info: Assert.True() Failure
info: Expected: True
info: Actual:   False
info:    at System.Net.WebSockets.Client.Tests.SendReceiveTest.SendReceive_Concurrent_Success(Uri server) in C:\Development\dotnet\runtime\src\libraries\System.Net.WebSockets.Client\tests\SendReceiveTest.cs:line 418
info: --- End of stack trace from previous location ---
info: System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest 0.1870299 ms
info: Received expected 2428 of C:\Development\dotnet\runtime\artifacts\bin\System.Net.WebSockets.Client.Tests\Debug\net9.0-browser\browser-wasm\AppBundle\xharness-output\testResults.xml
info: Finished writing 2428 bytes of RESULTXML
info: Xml file was written to the provided writer.
info: Tests run: 248 Passed: 0 Inconclusive: 0 Failed: 2 Ignored: 246 Skipped: 0
info: test-main.js exiting WasmTestRunner.dll System.Net.WebSockets.Client.Tests.dll with result 1
info: WASM EXIT 1
info: Waiting to flush log messages with a timeout of 120 secs ..
fail: Application has finished with exit code TESTS_FAILED but 0 was expected

@maraf maraf added the wasm WASM area label Feb 2, 2024
@maraf maraf self-assigned this Feb 2, 2024
@akoeplinger akoeplinger merged commit 23bffcf into dotnet:main Feb 6, 2024
15 of 17 checks passed
@maraf maraf deleted the WasmRunnerThreads2 branch February 6, 2024 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wasm WASM area
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants