Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mono] Tests failing with Failed to find compatible device: arm64-v8a #87267

Closed
ivanpovazan opened this issue Jun 8, 2023 · 7 comments
Closed
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab

Comments

@ivanpovazan
Copy link
Member

ivanpovazan commented Jun 8, 2023

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error leg:

  • linux_bionic-arm64 Release AllSubsets_Mono

Tests failing:

  • Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
  • Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
  • System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
  • System.IO.Pipes.Tests.WorkItemExecution
  • System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: Failed to find compatible device: arm64-v8a",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
476263 dotnet/runtime Microsoft.VisualBasic.Core.Tests.WorkItemExecution
476004 dotnet/runtime Microsoft.XmlSerializer.Generator.Tests.WorkItemExecution
472380 dotnet/runtime System.ComponentModel.TypeConverter.Tests.WorkItemExecution #94844
475900 dotnet/runtime Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
475292 dotnet/runtime Microsoft.Extensions.Configuration.Binder.Tests.WorkItemExecution
475327 dotnet/runtime IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution
475076 dotnet/runtime Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution
474832 dotnet/runtime System.Composition.AttributeModel.Tests.WorkItemExecution
471584 dotnet/runtime System.Threading.ThreadPool.Tests.WorkItemExecution
471508 dotnet/runtime System.Text.RegularExpressions.Unit.Tests.WorkItemExecution
471393 dotnet/runtime System.Xml.RW.CustomReader.Tests.WorkItemExecution
471313 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution
470965 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution #93220
470621 dotnet/runtime System.Runtime.Tests.WorkItemExecution
470544 dotnet/runtime System.Security.Cryptography.Xml.Tests.WorkItemExecution
470317 dotnet/runtime System.Security.Cryptography.Xml.Tests.WorkItemExecution
470325 dotnet/runtime System.Xml.Linq.Axes.Tests.WorkItemExecution
469832 dotnet/runtime System.Runtime.Serialization.Json.ReflectionOnly.Tests.WorkItemExecution
469799 dotnet/runtime System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
469455 dotnet/runtime System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
469234 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution
468967 dotnet/runtime System.Xml.RW.SubtreeReader.Tests.WorkItemExecution
468943 dotnet/runtime System.Security.Cryptography.ProtectedData.Tests.WorkItemExecution
468844 dotnet/runtime System.Xml.Linq.Misc.Tests.WorkItemExecution
468831 dotnet/runtime System.Threading.Tasks.Extensions.Tests.WorkItemExecution
468599 dotnet/runtime System.Threading.Tasks.Tests.WorkItemExecution
468590 dotnet/runtime System.Xml.RW.NameTable.Tests.WorkItemExecution
468420 dotnet/runtime System.Threading.Channels.Tests.WorkItemExecution
468095 dotnet/runtime System.Xml.RW.CustomReader.Tests.WorkItemExecution
467978 dotnet/runtime System.ValueTuple.Tests.WorkItemExecution
467759 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #94602
467535 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution
467341 dotnet/runtime System.Threading.Channels.Tests.WorkItemExecution
467123 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution
466947 dotnet/runtime System.Web.HttpUtility.Tests.WorkItemExecution
466956 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution
466670 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
466386 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
466152 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
462819 dotnet/runtime Microsoft.Extensions.Http.Tests.WorkItemExecution
462611 dotnet/runtime System.CodeDom.Tests.WorkItemExecution
462515 dotnet/runtime System.CodeDom.Tests.WorkItemExecution
462059 dotnet/runtime Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
461730 dotnet/runtime Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
458877 dotnet/runtime Microsoft.XmlSerializer.Generator.Tests.WorkItemExecution
458245 dotnet/runtime System.Diagnostics.Contracts.Tests.WorkItemExecution
457943 dotnet/runtime Microsoft.Extensions.Configuration.EnvironmentVariables.Tests.WorkItemExecution
457916 dotnet/runtime Microsoft.Extensions.Configuration.Json.Tests.WorkItemExecution #94274
457319 dotnet/runtime System.ComponentModel.EventBasedAsync.Tests.WorkItemExecution
456975 dotnet/runtime Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
456789 dotnet/runtime System.AppContext.Tests.WorkItemExecution
456721 dotnet/runtime System.Buffers.Tests.WorkItemExecution #94038
456534 dotnet/runtime Microsoft.Extensions.Options.Tests.WorkItemExecution
456177 dotnet/runtime System.Threading.Tasks.Extensions.Tests.WorkItemExecution
455954 dotnet/runtime System.Threading.Tests.WorkItemExecution
455930 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
455771 dotnet/runtime System.Runtime.Serialization.Schema.Tests.WorkItemExecution
455739 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
455556 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution
455375 dotnet/runtime System.Text.Encoding.Extensions.Tests.WorkItemExecution #94038
454951 dotnet/runtime System.Threading.Overlapped.Tests.WorkItemExecution
454774 dotnet/runtime System.Threading.RateLimiting.Tests.WorkItemExecution
454509 dotnet/runtime System.Threading.Channels.Tests.WorkItemExecution
454434 dotnet/runtime System.Threading.Tasks.Extensions.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 8 64

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: crit: Failed to find compatible device: arm64-v8a
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:22:27 PM UTC

@ivanpovazan ivanpovazan added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 8, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 8, 2023
@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055
Build error leg or test failing:

  • Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
  • Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
  • System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
  • System.IO.Pipes.Tests.WorkItemExecution
  • System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "ERROR: The process \"corerun.exe\" not found",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
Author: ivanpovazan
Assignees: -
Labels:

area-Infrastructure-libraries, blocking-clean-ci, Known Build Error

Milestone: -

@ivanpovazan ivanpovazan changed the title [mono] Tests failing on linux_bionic-arm64 Release AllSubsets_Mono with corerun.exe not found [mono] Tests failing with corerun.exe not found Jun 8, 2023
@ivanpovazan ivanpovazan changed the title [mono] Tests failing with corerun.exe not found Tests failing with corerun.exe not found Jun 8, 2023
@jakobbotsch
Copy link
Member

jakobbotsch commented Jun 8, 2023

@ivanpovazan I think this log statement is from

<HelixPostCommand Include="taskkill.exe /f /im corerun.exe"/>

and is harmless. E.g. it triggered on my #87165 for the JIT.opt workitem but the actual failure there was #87258

@ivanpovazan
Copy link
Member Author

@jakobbotsch thanks for the info!
Does it mean that the issue goes away on retries?

@jakobbotsch
Copy link
Member

I don't think there is an issue related to that log statement and I don't think it causes the work items to fail. So you should look for other failures in the logs. E.g. in
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-86652-merge-1c9c9cfd4ebd4ec7b3/System.IO.Pipes.Tests/1/console.5b1e5bba.log?helixlogtype=result
(for the System.IO.Pipelines failure in net8.0-linux-Release-arm64-Mono_Release_LinuxBionic-Windows.10.Amd64.Android.Open), the log ends with "ERROR: The process "corerun.exe" not found.", but the actual failure is above that:

[02:00:43] info: An error occurred running System.IO.Pipes.Tests.sh
[02:00:43] dbug: Exit code: 1
                 Std out:
                   Discovering: System.IO.Pipes.Tests (method display = ClassAndMethod, method display options = None)
                   Discovered:  System.IO.Pipes.Tests (found 107 of 538 test cases)
                   Starting:    System.IO.Pipes.Tests (parallel test collections = on, max threads = 8)
                       System.UnauthorizedAccessException : Access to the path '/data/local/tmp/CoreFxPipe_g2n3xo52.px0' is denied.
                       Stack Trace:
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(336,0): at System.IO.Pipes.NamedPipeServerStream.SharedServer..ctor(String path, Int32 maxCount, Boolean isFirstPipeInstance)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(279,0): at System.IO.Pipes.NamedPipeServerStream.SharedServer.Get(String path, Int32 maxCount, PipeOptions pipeOptions)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(43,0): at System.IO.Pipes.NamedPipeServerStream.Create(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options, Int32 inBufferSize, Int32 outBufferSize, HandleInheritability inheritability)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.cs(82,0): at System.IO.Pipes.NamedPipeServerStream..ctor(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options, Int32 inBufferSize, Int32 outBufferSize, HandleInheritability inheritability)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.cs(39,0): at System.IO.Pipes.NamedPipeServerStream..ctor(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options)
                         /_/src/libraries/System.IO.Pipes/tests/NamedPipeTests/NamedPipeTest.CreateServer.cs(263,0): at System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException()
                         /_/src/mono/System.Private.CoreLib/src/System/Reflection/MethodInvoker.Mono.cs(30,0): at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, IntPtr* args)
                         /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(59,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
                   Finished:    System.IO.Pipes.Tests
                 === TEST EXECUTION SUMMARY ===
                    System.IO.Pipes.Tests  Total: 150, Errors: 0, Failed: 1, Skipped: 0, Time: 2.364s
                 
                 
                 Std err:
                     System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL]

So for example you could use System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL] as the error message in the known error template. Using "ERROR: The process "corerun.exe" not found",` is causing tons of unrelated PRs to be linked to this issue.

@ivanpovazan ivanpovazan changed the title Tests failing with corerun.exe not found [mono] Tests failing with Failed to find compatible device: arm64-v8a Jun 8, 2023
@ivanpovazan
Copy link
Member Author

ivanpovazan commented Jun 8, 2023

Thank you very much for the detailed explanation, and sorry for the confusion.

I have updated the search term to be: crit: Failed to find compatible device: arm64-v8a instead of wrongly accused corerun.exe not found

This issue seems to be related to a missing device on linux_bionic-arm64 Release AllSubsets_Mono, while on the linux_bionic-x64 Release AllSubsets_Mono (note the difference in architecture) there is the failing test @jakobbotsch noted above:

System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL]

I will create a separate tracking issue as these don't seem to be related.

@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error leg:

  • linux_bionic-arm64 Release AllSubsets_Mono

Tests failing:

  • Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
  • Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
  • System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
  • System.IO.Pipes.Tests.WorkItemExecution
  • System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: Failed to find compatible device: arm64-v8a",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
299742 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299732 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299717 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ServerCertificates_Test.NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds
299738 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299736 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299735 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299727 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299719 dotnet/runtime System.Text.RegularExpressions.Tests.RegexMatchTests.StressTestDeepNestingOfLoops
299731 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
299527 dotnet/runtime Workloads-NoWebcil-Wasm.Build.Tests.Blazor.MiscTests.WorkItemExecution #86546
295712 dotnet/runtime chrome-DebuggerTests.WorkItemExecution #86934
299396 dotnet/runtime JIT.opt.WorkItemExecution
299429 dotnet/runtime JIT.opt.WorkItemExecution
299427 dotnet/runtime profiler.rejit.WorkItemExecution
299459 dotnet/runtime BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover
299397 dotnet/runtime JIT.opt.WorkItemExecution
299370 dotnet/runtime JIT.opt.WorkItemExecution
299406 dotnet/runtime nativeaot.SmokeTests.WorkItemExecution #87148
299242 dotnet/runtime profiler.rejit.WorkItemExecution
299286 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #87241
299094 dotnet/runtime JIT.opt.WorkItemExecution #87165
298055 dotnet/runtime System.Linq.Queryable.Tests.WorkItemExecution #86652
298971 dotnet/runtime chrome-DebuggerTests.WorkItemExecution #87232
298972 dotnet/runtime chrome-DebuggerTests.WorkItemExecution #87233
299037 dotnet/runtime System.Net.Http.Functional.Tests.SocksProxyTest_Http2.TestLoopbackAsync
298965 dotnet/runtime BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover #87232
298917 dotnet/runtime System.Data.OleDb.Tests.WorkItemExecution #87229
298677 dotnet/runtime profiler.rejit.WorkItemExecution
298816 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http3.ContentLength_DoesNotMatchRequestContentLength_Throws
298813 dotnet/runtime WasmTestOnBrowser-simd-System.Runtime.Intrinsics.Tests.WorkItemExecution
298779 dotnet/runtime ComInterfaceGenerator.Unit.Tests.ComHostingDoesNotSupportGeneratedComInterfaceTests.ComVisibleType_ComImportInterfacesOnly_DoesNotReportDiagnostic #87223
297507 dotnet/runtime WasmTestOnBrowser-System.Runtime.Loader.Tests.WorkItemExecution #85796
298658 dotnet/runtime System.Text.Json.SourceGeneration.UnitTests.JsonSourceGeneratorDiagnosticsTests.JsonStringEnumConverterWarns #87149
298623 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection #86759
298540 dotnet/runtime System.Runtime.Tests.WorkItemExecution #86875
298472 dotnet/runtime System.Numerics.Tests.Matrix4x4Tests.Matrix4x4CreateShadowTest02 #87126
297988 dotnet/runtime Wasm.Build.NativeRebuild.Tests.NoopNativeRebuildTest.NoOpRebuildForNativeBuilds #86255
298474 dotnet/runtime System.Text.RegularExpressions.Tests.WorkItemExecution #87126
298200 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298206 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298223 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http3_MsQuic.Public_Interop_Upgrade_Success
298221 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298203 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http3_MsQuic.Public_Interop_Upgrade_Success
298218 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298213 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298209 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298210 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298219 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298199 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298211 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
298224 dotnet/runtime PayloadGroup0.WorkItemExecution
298226 dotnet/runtime System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Test.NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds
298124 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws #87162
297984 dotnet/runtime Workloads-Wasm.Build.Tests.Blazor.BuildPublishTests.WorkItemExecution #87051
298068 dotnet/runtime System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
298000 dotnet/runtime JIT/SIMD/VectorAdd_ro/VectorAdd_ro.dll
297910 dotnet/runtime profiler.rejit.WorkItemExecution
297843 dotnet/runtime JIT.opt.WorkItemExecution
297912 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http2.ThresholdExceeded_ThrowsException #87203

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
40 59 59
Author: ivanpovazan
Assignees: -
Labels:

blocking-clean-ci, untriaged, area-Infrastructure-mono, Known Build Error

Milestone: -

@akoeplinger
Copy link
Member

The recent failures were because of a broken machine: DNCENGWIN-038
It was fixed recently (dotnet/dnceng#1383 (comment)) so closing this.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 24, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Dec 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Development

No branches or pull requests

3 participants