Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for "WORKLOAD TIMED OUT" #90309

Closed
steveharter opened this issue Aug 10, 2023 · 7 comments
Closed

Tracking issue for "WORKLOAD TIMED OUT" #90309

steveharter opened this issue Aug 10, 2023 · 7 comments
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-ios Apple iOS os-tvos Apple tvOS

Comments

@steveharter
Copy link
Member

steveharter commented Aug 10, 2023

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build Definition Test Pull Request
749807 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #105233
748937 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
748618 dotnet/runtime System.Numerics.Tensors.Net8.Tests.WorkItemExecution
748590 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution
748320 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #105167
747478 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
748081 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
747944 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #105153
747724 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #104962
747439 dotnet/runtime System.Numerics.Tensors.Net8.Tests.WorkItemExecution
746875 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #104856
746673 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #104848
746336 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
746053 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #103755
745979 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
745513 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
743450 dotnet/runtime JIT_opt.WorkItemExecution
743225 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
743222 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
740287 dotnet/runtime System.Numerics.Tensors.Net8.Tests.WorkItemExecution
739967 dotnet/runtime JIT_opt.WorkItemExecution
739577 dotnet/runtime Loader_classloader.WorkItemExecution
739539 dotnet/runtime Loader_classloader.WorkItemExecution
737515 dotnet/runtime JIT_opt.WorkItemExecution
735679 dotnet/runtime System.Numerics.Tensors.Net8.Tests.WorkItemExecution
735671 dotnet/runtime System.Runtime.Tests.WorkItemExecution #103585
735197 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
734188 dotnet/runtime System.Numerics.Tensors.Tests.WorkItemExecution
727973 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
728292 dotnet/runtime JIT_opt.WorkItemExecution
727276 dotnet/runtime System.Numerics.Tensors.Tests.WorkItemExecution
721722 dotnet/runtime System.ValueTuple.Tests.WorkItemExecution
720435 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102403

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 19 33
@steveharter steveharter added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Infrastructure Known Build Error Use this to report build issues in the .NET Helix tab labels Aug 10, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 10, 2023
@ghost
Copy link

ghost commented Aug 10, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command..",
  "BuildRetry": false
}
Author: steveharter
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, Known Build Error

Milestone: -

@ghost
Copy link

ghost commented Sep 14, 2023

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build Definition Test Pull Request
406029 dotnet/runtime Loader_classloader.WorkItemExecution #91261
406036 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
405860 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405505 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405371 dotnet/runtime JIT_opt.WorkItemExecution
405156 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405201 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91753
405175 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91913
404815 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91498
404502 dotnet/runtime JIT_opt.WorkItemExecution
403846 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
404006 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91964
403649 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
403432 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91945
403331 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #91910
403278 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
402917 dotnet/runtime JIT_opt.WorkItemExecution #91925
386099 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #90764
402796 dotnet/runtime JIT_opt.WorkItemExecution
402586 dotnet/runtime JIT_opt.WorkItemExecution
402622 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91915
402416 dotnet/runtime JIT_opt.WorkItemExecution
402225 dotnet/runtime JIT_opt.WorkItemExecution
402136 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
402090 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #90436
401838 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
401993 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91892
398263 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91690
401505 dotnet/runtime JIT_opt.WorkItemExecution #91042
401473 dotnet/runtime JIT_opt.WorkItemExecution
401430 dotnet/runtime JIT_opt.WorkItemExecution #91788
401363 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91671
401270 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
401274 dotnet/runtime JIT_opt.WorkItemExecution #91808
401015 dotnet/runtime JIT_opt.WorkItemExecution
400995 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91482
400925 dotnet/runtime JIT_opt.WorkItemExecution #91831
400893 dotnet/runtime JIT_opt.WorkItemExecution
400630 dotnet/runtime JIT_opt.WorkItemExecution
400465 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91828
400257 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
400363 dotnet/runtime JIT_opt.WorkItemExecution #91468
400157 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91753
399738 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91790
399748 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91792
399536 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91785
399499 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
399419 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91774
399275 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
399203 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91713
398919 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398113 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91659
398621 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91520
398510 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398372 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91019
398117 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398093 dotnet/runtime JIT_opt.WorkItemExecution #91714
398051 dotnet/runtime JIT_opt.WorkItemExecution #91520
398047 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91520
397851 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
397767 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91703
397525 dotnet/runtime JIT_opt.WorkItemExecution
397329 dotnet/runtime JIT_opt.WorkItemExecution
397233 dotnet/runtime JIT_opt.WorkItemExecution #91684
397052 dotnet/runtime JIT_opt.WorkItemExecution #91458
397028 dotnet/runtime JIT_opt.WorkItemExecution #91290
396987 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91290
396849 dotnet/runtime JIT_opt.WorkItemExecution #91019
396755 dotnet/runtime JIT_opt.WorkItemExecution #91660
396634 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396691 dotnet/runtime JIT_opt.WorkItemExecution
396584 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
396520 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396111 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396177 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91290
395791 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
395609 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
395438 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
395416 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
395450 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91536
395268 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
395186 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394754 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91116
394959 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91536
394722 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
394654 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394471 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394475 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91524
394436 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91520
394347 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394281 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394161 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91502
394046 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91497
393987 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393980 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393667 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393578 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393552 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91469
393434 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393189 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
Displaying 100 of 246 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 55 246
Author: steveharter
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, untriaged, os-ios, os-tvos, Known Build Error

Milestone: -

@steveisok
Copy link
Member

This is a variation of #82637

@agocke agocke removed this from AppModel Dec 2, 2023
@steveisok
Copy link
Member

steveisok commented Dec 10, 2023

@akoeplinger something is wrong w/ our classification when there is a crash in a test suite.

Check out https://dev.azure.com/dnceng-public/public/_build/results?buildId=494600&view=ms.vss-test-web.build-test-results-tab&runId=11481726&resultId=179767 for example. The net.dot log is huge (350MB) and seems to be logging a crash over and over again.

@akoeplinger
Copy link
Member

akoeplinger commented Dec 11, 2023

The classification itself is correct, we're timing out because of the crash log loop. I see a potential reason for this, discussing it on Discord.

Copy link
Contributor

Tagging subscribers to this area: @directhex, @matouskozak
See info in area-owners.md if you want to be subscribed.

@agocke agocke removed this from Runtime Infra Jul 8, 2024
@matouskozak
Copy link
Member

Closing this issue because the error message is too general and is matching all kinds of failures (see dotnet/xharness#1188).

@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Jul 22, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-ios Apple iOS os-tvos Apple tvOS
Projects
None yet
Development

No branches or pull requests

6 participants