Test failure: System.IO.FileSystem.Tests

[BruceForstall]

net8.0-windows-Release-x64-CoreCLR_checked-fullpgo_methodprofiling-Windows.10.Amd64.Open

https://dev.azure.com/dnceng-public/public/_build/results?buildId=320072&view=ms.vss-test-web.build-test-results-tab&runId=6562686&paneView=debug

C:\h\w\AA700969\w\BEE60A43\e>set DOTNET 
DOTNET_JitDelegateProfiling=1
DOTNET_JitVTableProfiling=1
DOTNET_ReadyToRun=0
DOTNET_TC_QuickJitForLoops=1
DOTNET_TieredCompilation=1
DOTNET_TieredPGO=1

C:\h\w\AA700969\w\BEE60A43\e>call RunTests.cmd --runtime-path C:\h\w\AA700969\p 
----- start Sun 06/25/2023 17:12:31.35 ===============  To repro directly: =====================================================
pushd C:\h\w\AA700969\w\BEE60A43\e\
"C:\h\w\AA700969\p\dotnet.exe" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\AA700969\w\BEE60A43\e>"C:\h\w\AA700969\p\dotnet.exe" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.IO.FileSystem.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.FileSystem.Tests (found 3755 of 4193 test cases)
  Starting:    System.IO.FileSystem.Tests (parallel test collections = on, max threads = 4)
...
Process terminated. Assertion failed.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.OverlappedValueTaskSource.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.OverlappedValueTaskSource.Windows.cs:line 193
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
----- end Sun 06/25/2023 17:14:02.96 ----- exit code -2146232797 ----------------------------------------------------------

@AndyAyersMS @EgorBo

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

net8.0-windows-Release-x64-CoreCLR_checked-fullpgo_methodprofiling-Windows.10.Amd64.Open

https://dev.azure.com/dnceng-public/public/_build/results?buildId=320072&view=ms.vss-test-web.build-test-results-tab&runId=6562686&paneView=debug

C:\h\w\AA700969\w\BEE60A43\e>set DOTNET 
DOTNET_JitDelegateProfiling=1
DOTNET_JitVTableProfiling=1
DOTNET_ReadyToRun=0
DOTNET_TC_QuickJitForLoops=1
DOTNET_TieredCompilation=1
DOTNET_TieredPGO=1

C:\h\w\AA700969\w\BEE60A43\e>call RunTests.cmd --runtime-path C:\h\w\AA700969\p 
----- start Sun 06/25/2023 17:12:31.35 ===============  To repro directly: =====================================================
pushd C:\h\w\AA700969\w\BEE60A43\e\
"C:\h\w\AA700969\p\dotnet.exe" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\AA700969\w\BEE60A43\e>"C:\h\w\AA700969\p\dotnet.exe" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.IO.FileSystem.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.FileSystem.Tests (found 3755 of 4193 test cases)
  Starting:    System.IO.FileSystem.Tests (parallel test collections = on, max threads = 4)
...
Process terminated. Assertion failed.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.OverlappedValueTaskSource.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.OverlappedValueTaskSource.Windows.cs:line 193
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
----- end Sun 06/25/2023 17:14:02.96 ----- exit code -2146232797 ----------------------------------------------------------

@AndyAyersMS @EgorBo

Author:	BruceForstall
Assignees:	AndyAyersMS
Labels:	os-windows, JitStress, arch-x64, area-CodeGen-coreclr, blocking-clean-ci-optional
Milestone:	8.0.0

[EgorBo]

So far happened only once in runtime-coreclr libraries-pgo and looks more like some IO/infra issue to me?

[BruceForstall]

It will run Saturday & Sunday, for additional data points. Also, there's a dump file.

[AndyAyersMS]

This didn't fail over the weekend and I can't repro. I'll look at the dump but it smells like some rare timing issue in in the test harness.

[AndyAyersMS]

The assertion hit was

runtime/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.OverlappedValueTaskSource.Windows.cs

Line 193 in f11edcd

Debug.Assert(numBytes < int.MaxValue);

Per debugger the value for numBytes was 0x00000000_fb339018 which is bigger than int.MaxValue = 0x7fffffff.

From the dump, the error code was ERROR_OPERATION_ABORTED = 0x3E3. Comments in the source nearby indicate that for this error code perhaps numBytes is not trustworthy.

@stephentoub any chance this assert is too aggressive?

[stephentoub]

cc: @adamsitnik

[AndyAyersMS]

@adamsitnik did you get a chance to look into this?

[AndyAyersMS]

Since we haven't seen this happen again I'm going to push this to future.

@adamsitnik take a look when you can.

[MichalStrehovsky]

I've ran into this in run-off-the-mill PR testing and filed it as #99517 using the build analysis template. This bug is pretty stale and has no traction. Maybe we can close it in favor of the one from template.

[stephentoub]

at Microsoft.Win32.SafeHandles.SafeFileHandle.OverlappedValueTaskSource.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOverlapped) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.OverlappedValueTaskSource.Windows.cs:line 193

Debug.Assert(numBytes < int.MaxValue); in the IO callback is an interesting assert to be failing.

[AndyAyersMS]

I've ran into this in run-off-the-mill PR testing and filed it as #99517 using the build analysis template. This bug is pretty stale and has no traction. Maybe we can close it in favor of the one from template.

Agreed, closing.