Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various WASM timeouts on CI #85304

Closed
pavelsavara opened this issue Apr 25, 2023 · 5 comments · Fixed by #85313
Closed

Various WASM timeouts on CI #85304

pavelsavara opened this issue Apr 25, 2023 · 5 comments · Fixed by #85313
Assignees
@pavelsavara
Labels
area-Build-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone
8.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Apr 25, 2023 •
edited by build-analysis bot
Loading

Error Blob

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "timed out after",
  "ExcludeConsoleLog": false
}

Reproduction Steps

No response

Report

Build Definition Step Name Console log Pull Request
255110 dotnet/runtime android-x64 Release AllSubsets_Mono_RuntimeTests minijit Log
254904 dotnet/runtime EnterpriseLinuxTests Log #85313
254662 dotnet/runtime EnterpriseLinuxTests Log #84793
254150 dotnet/runtime EnterpriseLinuxTests Log #85447
253140 dotnet/runtime EnterpriseLinuxTests Log #85313
250074 dotnet/runtime EnterpriseLinuxTests Log #85244
Build Definition Test Pull Request
255566 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #83157
255495 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #84494
255405 dotnet/runtime Workloads-Webcil-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution #85498
255005 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85283
255464 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85484
255472 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
255434 dotnet/runtime chrome-DebuggerTests.WorkItemExecution
254760 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85328
255350 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85444
255359 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85497
255261 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85474
255198 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85491
255188 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85459
255175 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85419
255214 dotnet/runtime System.Net.Requests.Tests.WorkItemExecution
255053 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85446
255075 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85373
255093 dotnet/runtime chrome-DebuggerTests.WorkItemExecution
255049 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85404
254725 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85470
255027 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85431
254972 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #84676
255000 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85282
255042 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution
254964 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85480
254986 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85482
255009 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
254913 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #84793
254936 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85284
254931 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85478
254898 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85424
254845 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85410
254835 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85474
254796 dotnet/runtime WasmTestOnBrowser-simd-System.Runtime.Intrinsics.Tests.WorkItemExecution
254779 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85254
254734 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85022
254681 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution
254266 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85307
254668 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85466
254633 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #84494
254629 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85458
254599 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85463
254595 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #83411
254554 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85384
254550 dotnet/runtime WasmTestOnNodeJS-System.Net.WebSockets.Client.Tests.WorkItemExecution #85384
254544 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85447
254533 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85461
254512 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85310
254526 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85460
254468 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85405
254460 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85459
254396 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254399 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254398 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254054 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85433
254387 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254400 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254407 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254388 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254410 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
254375 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85438
254371 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85437
254337 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85457
251755 dotnet/runtime WasmTestOnBrowser-simd-System.Runtime.Intrinsics.Tests.WorkItemExecution #85337
254323 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85254
254308 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85454
254235 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85405
254279 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85143
254274 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85143
254256 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
254240 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85402
254225 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85447
254145 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85446
254127 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85444
254131 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85445
254123 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85443
254102 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85438
254098 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85437
253680 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85310
254073 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85431
254082 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
254002 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution
253912 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85424
253894 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution
253826 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85283
253464 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85404
253831 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #84793
253809 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85328
253729 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85418
253756 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #83854
253779 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
253760 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution
253654 dotnet/runtime chrome-DebuggerTests.WorkItemExecution
253577 dotnet/runtime WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution #85410
253561 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85409
253537 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85308
253522 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85405
253471 dotnet/runtime chrome-DebuggerTests.WorkItemExecution #85352
253493 dotnet/runtime normal-System.Collections.Immutable.Tests.WorkItemExecution #85343
253445 dotnet/runtime Workloads-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution #83411
Displaying 100 of 264 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
74 270 270
@pavelsavara pavelsavara added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Apr 25, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Apr 25, 2023
@pavelsavara pavelsavara self-assigned this Apr 25, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 25, 2023
@pavelsavara pavelsavara mentioned this issue Apr 25, 2023
Closed
@pavelsavara pavelsavara added area-Build-mono and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 25, 2023
@pavelsavara pavelsavara mentioned this issue Apr 25, 2023
Merged
This was referenced Apr 25, 2023
Merged
Merged
@EgorBo EgorBo mentioned this issue Apr 26, 2023
Merged
@pavelsavara
Copy link
Member Author

pavelsavara commented Apr 26, 2023 •
edited
Loading

This is interesting interp on chrome Log

There test at [19:02:48] which takes over 3 minutes!
@kg could you please have look ? Is it bad test or we have some real trouble ?
cc @lewing

[19:02:24] info: [2023-04-25T19:02:24.696Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: Basic, jsonString: "{\n   \"age\" : 30,\n   \"first\" : \"John\",\n   "...)
[19:02:24] info: [2023-04-25T19:02:24.703Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: BasicLargeNum, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:02:48] info: [2023-04-25T19:02:48.258Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: BasicLargeNum, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:02:48] info: [2023-04-25T19:02:48.265Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: FullSchema1, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:07:01] info: [2023-04-25T19:07:01.237Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: FullSchema1, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:07:01] info: [2023-04-25T19:07:01.245Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: HelloWorld, jsonString: "{ \"message\": \"Hello, World!\" }")

@pavelsavara pavelsavara mentioned this issue Apr 26, 2023
Closed
@pavelsavara
Copy link
Member Author

pavelsavara commented Apr 26, 2023 •
edited
Loading

This log is example of V8 dying in the middle of some promise chain as reported here. #83443
Note lack of XUnit summary.

[11:27:41] info: [STRT] System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.WriteAsyncEnumerable_CancellationToken_IsPassedToAsyncEnumerator
[11:27:41] info: Process v8 exited with 0
                 
[11:27:41] info: Waiting to flush log messages with a timeout of 120 secs ..
[11:27:41] info: Application has finished with exit code: 0
XHarness exit code: 0

@pavelsavara
Copy link
Member Author

pavelsavara commented Apr 26, 2023 •
edited
Loading

This log is example where the NodeJS died in the middle of the test.

[11:37:13] info: Using js engine **NodeJS** from path /home/helixbot/node-v16.19.0-linux-x64/bin/node
...
[11:48:50] info: [STRT] System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.WriteAsyncEnumerableOfAsyncEnumerables<Int32>(source: [], delayInterval: 0, bufferSize: 1)
[11:48:50] info: [STRT] System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.WriteAsyncEnumerableOfAsyncEnumerables<Int32>(source: [0, 1, 2, 3, 4, ...], delayInterval: 0, bufferSize: 1)
[11:48:50] info: [STRT] System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.WriteAsyncEnumerableOfAsyncEnumerables<Int32>(source: [0, 1, 2, 3, 4, ...], delayInterval: 20, bufferSize: 20)
[11:48:50] info: Process node exited with 0
                 
[11:48:50] info: Waiting to flush log messages with a timeout of 120 secs ..
[11:48:50] info: Application has finished with exit code: 0
XHarness exit code: 0
/root/helix/work/workitem/e /root/helix/work/workitem/e
+ export _commandExitCode=0
+ python -u /root/helix/work/workitem/u/xharness-event-processor.py

@kg
Copy link
Member

kg commented Apr 27, 2023

This is interesting interop on chrome Log

There test at [19:02:48] which takes over 3 minutes! @kg could you please have look ? Is it bad test or we have some real trouble ? cc @lewing

[19:02:24] info: [2023-04-25T19:02:24.696Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: Basic, jsonString: "{\n   \"age\" : 30,\n   \"first\" : \"John\",\n   "...)
[19:02:24] info: [2023-04-25T19:02:24.703Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: BasicLargeNum, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:02:48] info: [2023-04-25T19:02:48.258Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: BasicLargeNum, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:02:48] info: [2023-04-25T19:02:48.265Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: FullSchema1, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:07:01] info: [2023-04-25T19:07:01.237Z] [PASS] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: FullSchema1, jsonString: "{\"age\":30,\"first\":\"John\",\"last\":\"Smith\","...)
[19:07:01] info: [2023-04-25T19:07:01.245Z] [STRT] System.Text.Json.Tests.Utf8JsonReaderTests.TestPartialJsonReader(compactData: False, type: HelloWorld, jsonString: "{ \"message\": \"Hello, World!\" }")

I can't find the actual definition of FullSchema1 (it appears to reference a FullJsonSchema1, but I can't find any references to that name in the repo or via a google search). Does anyone know what these tests are doing?

@kg
Copy link
Member

kg commented Apr 27, 2023

OK, I found the definition - it's a string resource in a resx file, and github search is broken and doesn't see it. The value is this:

{"age":30,"first":"John","last":"Smith","phoneNumbers":["425-000-1212","425-000-1213"],"address":{"street":"1MicrosoftWay","city":"Redmond","zip":98052},"IDs":[425,-70,9223372036854775000],"arrayWithObjects":["text",14,[],null,false,{},{"time":24},["1","2","3"]],"boolean":false,"null":null,"objectName":{"group":{"array":[false],"field":"simple","anotherFieldNum":5,"anotherFieldBool":true,"lastField":null}},"emptyObject":{}}

I can't see any reason why this would take multiple seconds, it's a very small chunk of data. Maybe the time is a GC pause caused by the tests previously doing tons of allocations? The JSON infra may also be allocating delegates for field access which could cause us to spend time generating interp and jiterp code, but that also shouldn't take more than dozens of ms at most.

@ManickaP ManickaP mentioned this issue Apr 27, 2023
Closed
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Apr 27, 2023
@pavelsavara pavelsavara changed the title Various timeouts on CI Various WASM timeouts on CI Apr 27, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Apr 28, 2023
@CarnaViire CarnaViire mentioned this issue May 4, 2023
Merged
@ghost ghost locked as resolved and limited conversation to collaborators May 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@pavelsavara pavelsavara

Labels
area-Build-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

[browser] Expanding timeouts for long running test projects pavelsavara/runtime
2 participants
@kg @pavelsavara