Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sporadic failure in file enumeration tests on Unix #54589

Closed
danmoseley opened this issue Jun 23, 2021 · 9 comments · Fixed by #54660
Closed

Sporadic failure in file enumeration tests on Unix #54589

danmoseley opened this issue Jun 23, 2021 · 9 comments · Fixed by #54660
Assignees
@adamsitnik
Labels
area-System.IO blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' bug
Milestone
6.0.0

Comments

@danmoseley
Copy link
Member

Examples - -

Ubuntu Arm

    System.IO.Tests.DirectoryInfo_EnumFSI_str_str.SearchPatternCaseSensitive [FAIL]
      System.UnauthorizedAccessException : Access to the path '/tmp/DirectoryInfo_EnumFSI_str_str_i42flluh.s3p/SearchPatternCaseSensitive_705_652a74c1' is denied.
      ---- System.IO.IOException : Bad file descriptor
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(197,0): at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(110,0): at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
        /_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs(27,0): at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
        /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/EnumerableAPIs.cs(73,0): at System.IO.Tests.DirectoryInfo_EnumFSI_str_str.GetEntries(String path, String searchPattern)
        /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs(719,0): at System.IO.Tests.Directory_GetFileSystemEntries_str_str.SearchPatternCaseSensitive()
        ----- Inner Stack Trace -----

osx 10.13 amd64

    System.IO.Tests.DirectoryInfo_EnumerateFiles_str_str.PatternTests_DosDotTrailingDot(pattern: "foo.*", sourceFiles: ["foo", "foo.", "foo.t", "foo.tx", "foo.txt", ...], expected: ["foo", "foo.", "foo.t", "foo.tx", "foo.txt", ...]) [FAIL]
      System.UnauthorizedAccessException : Access to the path '/var/folders/fs/wj3by_p90wq589rz5xk5ltb40000gy/T/DirectoryInfo_EnumerateFiles_str_str_mmyqz5al.lod/PrepareDirectory_539_23baf34f' is denied.
      ---- System.IO.IOException : Bad file descriptor
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(178,0): at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(93,0): at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
        /_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs(27,0): at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
        /_/src/libraries/System.Linq/src/System/Linq/ToCollection.cs(11,0): at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
        /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/EnumerableAPIs.cs(28,0): at System.IO.Tests.DirectoryInfo_EnumerateFiles_str_str.GetEntries(String path, String searchPattern)
        /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs(343,0): at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected)
        ----- Inner Stack Trace -----

Some other callstack variations:

   at System.IO.Strategies.FileStreamHelpers.CheckFileCall(Int64 result, String path, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 29
   at System.IO.Strategies.Net5CompatFileStreamStrategy.CheckFileCall(Int32 result, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 624
   at System.IO.Strategies.Net5CompatFileStreamStrategy.WriteNative(ReadOnlySpan`1 source) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 457
   at System.IO.Strategies.Net5CompatFileStreamStrategy.FlushWriteBuffer(Boolean calledFromFinalizer) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 189
   at System.IO.Strategies.Net5CompatFileStreamStrategy.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 93
   at System.IO.Strategies.Net5CompatFileStreamStrategy.DisposeInternal(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.cs:line 110
   at System.IO.FileStream.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 476
   at System.IO.Stream.Close() in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 165
   at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 235
   at System.IO.StreamWriter.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 218
   at System.IO.TextWriter.Dispose() in /_/src/libraries/System.Private.CoreLib/src/System/IO/TextWriter.cs:line 77
   at System.IO.File.WriteAllText(String path, String contents) in /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs:line 312
   at System.IO.Tests.Directory_GetFileSystemEntries_str.CreateItem(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str.cs:line 28
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PrepareDirectory(String[] sourceFiles, Boolean useExtendedPaths) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 542
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 342
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 370
----- Inner Stack Trace -----

   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs:line 91
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs:line 158
   at System.IO.Strategies.Net5CompatFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.cs:line 92
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 16
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.cs:line 20
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 210
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 146
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 141
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 136
   at System.IO.StreamWriter.ValidateArgsAndOpenPath(String path, Boolean append, Encoding encoding, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 182
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 153
   at System.IO.StreamWriter..ctor(String path) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 138
   at System.IO.File.WriteAllText(String path, String contents) in /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs:line 309
   at System.IO.Tests.Directory_GetFileSystemEntries_str.CreateItem(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str.cs:line 28
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PrepareDirectory(String[] sourceFiles, Boolean useExtendedPaths) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 542
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 342
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 370
----- Inner Stack Trace -----

This began failing a few times a day on 6/15 which is coincident with #53669. Perhaps this will be fixed by #54483 @adamsitnik
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.IO untriaged New issue has not been triaged by the area owner labels Jun 23, 2021
@ghost
Copy link

ghost commented Jun 23, 2021

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Examples - -

Ubuntu Arm

    System.IO.Tests.DirectoryInfo_EnumFSI_str_str.SearchPatternCaseSensitive [FAIL]
      System.UnauthorizedAccessException : Access to the path '/tmp/DirectoryInfo_EnumFSI_str_str_i42flluh.s3p/SearchPatternCaseSensitive_705_652a74c1' is denied.
      ---- System.IO.IOException : Bad file descriptor
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(197,0): at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(110,0): at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
        /_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs(27,0): at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
        /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/EnumerableAPIs.cs(73,0): at System.IO.Tests.DirectoryInfo_EnumFSI_str_str.GetEntries(String path, String searchPattern)
        /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs(719,0): at System.IO.Tests.Directory_GetFileSystemEntries_str_str.SearchPatternCaseSensitive()
        ----- Inner Stack Trace -----

osx 10.13 amd64

    System.IO.Tests.DirectoryInfo_EnumerateFiles_str_str.PatternTests_DosDotTrailingDot(pattern: "foo.*", sourceFiles: ["foo", "foo.", "foo.t", "foo.tx", "foo.txt", ...], expected: ["foo", "foo.", "foo.t", "foo.tx", "foo.txt", ...]) [FAIL]
      System.UnauthorizedAccessException : Access to the path '/var/folders/fs/wj3by_p90wq589rz5xk5ltb40000gy/T/DirectoryInfo_EnumerateFiles_str_str_mmyqz5al.lod/PrepareDirectory_539_23baf34f' is denied.
      ---- System.IO.IOException : Bad file descriptor
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(178,0): at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
        /_/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs(93,0): at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
        /_/src/libraries/System.Linq/src/System/Linq/Select.SpeedOpt.cs(27,0): at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
        /_/src/libraries/System.Linq/src/System/Linq/ToCollection.cs(11,0): at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
        /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/EnumerableAPIs.cs(28,0): at System.IO.Tests.DirectoryInfo_EnumerateFiles_str_str.GetEntries(String path, String searchPattern)
        /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs(343,0): at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected)
        ----- Inner Stack Trace -----

Some other callstack variations:

   at System.IO.Strategies.FileStreamHelpers.CheckFileCall(Int64 result, String path, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 29
   at System.IO.Strategies.Net5CompatFileStreamStrategy.CheckFileCall(Int32 result, Boolean ignoreNotSupported) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 624
   at System.IO.Strategies.Net5CompatFileStreamStrategy.WriteNative(ReadOnlySpan`1 source) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 457
   at System.IO.Strategies.Net5CompatFileStreamStrategy.FlushWriteBuffer(Boolean calledFromFinalizer) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 189
   at System.IO.Strategies.Net5CompatFileStreamStrategy.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.Unix.cs:line 93
   at System.IO.Strategies.Net5CompatFileStreamStrategy.DisposeInternal(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.cs:line 110
   at System.IO.FileStream.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 476
   at System.IO.Stream.Close() in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 165
   at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 235
   at System.IO.StreamWriter.Dispose(Boolean disposing) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 218
   at System.IO.TextWriter.Dispose() in /_/src/libraries/System.Private.CoreLib/src/System/IO/TextWriter.cs:line 77
   at System.IO.File.WriteAllText(String path, String contents) in /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs:line 312
   at System.IO.Tests.Directory_GetFileSystemEntries_str.CreateItem(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str.cs:line 28
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PrepareDirectory(String[] sourceFiles, Boolean useExtendedPaths) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 542
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 342
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 370
----- Inner Stack Trace -----

   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs:line 91
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs:line 158
   at System.IO.Strategies.Net5CompatFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.cs:line 92
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs:line 16
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.cs:line 20
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 210
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 146
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 141
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs:line 136
   at System.IO.StreamWriter.ValidateArgsAndOpenPath(String path, Boolean append, Encoding encoding, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 182
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 153
   at System.IO.StreamWriter..ctor(String path) in /_/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs:line 138
   at System.IO.File.WriteAllText(String path, String contents) in /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs:line 309
   at System.IO.Tests.Directory_GetFileSystemEntries_str.CreateItem(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str.cs:line 28
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PrepareDirectory(String[] sourceFiles, Boolean useExtendedPaths) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 542
   at System.IO.Tests.Directory_GetFileSystemEntries_str_str.PatternTests_DosDotTrailingDot(String pattern, String[] sourceFiles, String[] expected) in /_/src/libraries/System.IO.FileSystem/tests/Directory/GetFileSystemEntries_str_str.cs:line 342
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 370
----- Inner Stack Trace -----

This began failing a few times a day on 6/15 which is coincident with #53669. Perhaps this will be fixed by #54483 @adamsitnik

Author: danmoseley
Assignees: -
Labels:

area-System.IO, untriaged
Milestone: -

@danmoseley danmoseley mentioned this issue Jun 23, 2021
Merged
@danmoseley
Copy link
Member Author

Doh - that's a Windows change, still the date these started seems to be 2021-06-15 21:36:40.9310000

https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAG2QQWsCMRCF74L%2fYdiLLay6emthC7YiWKjQKvQo02Tqxt2dSGYWEfzxTdT2VAhMMvky771sSPSDpGtUoN87w947htqxLR0zBfj0oV4qtQKe%2fw5L%2by%2f76r8uWKxLm4BjRYFg4dhJRRaeSrCopK6lu2kxLYbFw3ACxWOR1n2%2fh2xhczoQiGJQOTqtIFufJCqOVqSj7Iq8kVbeQllC9uKjsNHtnGW77owhkRt0zZSgwQJdM%2bj3xuPrYxHcERjPio4FshkDapQ4KBxRoEVLoB40IItTF%2fMgKEqdmgjfjrFJBpVSPAanUKEFbAKhPcW57aEhJZvcnuEQ%2fD4ahHVKRDaHeRcwDc1vMXJ476ijFbY0yX%2fd5Yk39SagiftZ2HUtsUq8R1M5vtA5PEeHpsovX%2fYDIyPQ4McBAAA%3d&web=0

TestResults
| join kind=inner WorkItems on WorkItemId
| join kind=inner Jobs on JobId
| where Finished >= datetime(2020-09-1 0:00:00)
and Type startswith "System.Net."
and Method == "Connect_Dns_Success"
and Result == 'Fail'
//and Message contains "An attempt was made to transition a task to a final state when it had already completed."
| project Started, Duration, Method, QueueName1, Message, StackTrace, Arguments, MachineName, Branch, Type

@danmoseley danmoseley mentioned this issue Jun 23, 2021
Closed
@SingleAccretion SingleAccretion mentioned this issue Jun 23, 2021
Merged
@stephentoub stephentoub added this to the 6.0.0 milestone Jun 23, 2021
@stephentoub stephentoub mentioned this issue Jun 23, 2021
Closed
@adamsitnik adamsitnik self-assigned this Jun 23, 2021
@adamsitnik adamsitnik removed the untriaged New issue has not been triaged by the area owner label Jun 23, 2021
@adamsitnik
Copy link
Member

The issue seems to be very mysterious. Let's consider following example:

        System.UnauthorizedAccessException : Access to the path '/tmp/Directory_GetFiles_str_str_so_u0xtgtez.eug/PrepareDirectory_539_628354c3/foobar.txt' is denied.
        ---- System.IO.IOException : Bad file descriptor
        Stack Trace:
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs(74,0): at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs(158,0): at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/Net5CompatFileStreamStrategy.cs(92,0): at System.IO.Strategies.Net5CompatFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.Unix.cs(16,0): at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.cs(20,0): at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs(210,0): at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs(146,0): at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
          /home/adam/projects/main/src/libraries/System.Private.CoreLib/src/System/IO/FileStream.cs(141,0): at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)

The source code:

runtime/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs

Lines 43 to 74 in 0416c34

SafeFileHandle handle = Interop.Sys.Open(path, flags, mode);
if (handle.IsInvalid)
{
handle.Dispose();
Interop.ErrorInfo error = Interop.Sys.GetLastErrorInfo();
// If we fail to open the file due to a path not existing, we need to know whether to blame
// the file itself or its directory. If we're creating the file, then we blame the directory,
// otherwise we blame the file.
//
// When opening, we need to align with Windows, which considers a missing path to be
// FileNotFound only if the containing directory exists.
bool isDirectory = (error.Error == Interop.Error.ENOENT) &&
((flags & Interop.Sys.OpenFlags.O_CREAT) != 0
|| !DirectoryExists(Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(path!))!));
Interop.CheckIo(
error.Error,
path,
isDirectory,
errorRewriter: e => (e.Error == Interop.Error.EISDIR) ? Interop.Error.EACCES.Info() : e);
}
// Make sure it's not a directory; we do this after opening it once we have a file descriptor
// to avoid race conditions.
Interop.Sys.FileStatus status;
if (Interop.Sys.FStat(handle, out status) != 0)
{
handle.Dispose();
throw Interop.GetExceptionForIoErrno(Interop.Sys.GetLastErrorInfo(), path);

What happens:

  1. The file is being opened and a SafeFileHandle is being created. The result is stored in a local variable, which is not shared with anyone else. (line 43)
  2. We check if handle is valid and it's fine (line 45)
  3. We call FStat using the handle and it fails. (line 71)
  4. The last error returns Bad file descriptor. Descriptor can be bad when it's closed. But how is it possible that it got closed since it has not been shared with anyone yet? (line 74)

My current reaction:
image

@danmoseley
Copy link
Member Author

Brainstorming - are there other possible causes of EBADF? Eg., file is locked by someone using a different fd?
@tmds

@tmds
Copy link
Member

tmds commented Jun 23, 2021

A thought about the code @adamsitnik shared:

 if (Interop.Sys.FStat(handle, out status) != 0) 
 { 
     handle.Dispose(); 
     throw Interop.GetExceptionForIoErrno(Interop.Sys.GetLastErrorInfo(), path);

We should call GetLastErrorInfo before calling Dispose, otherwise Dispose might change the error.

I'll look at the issue closer in the coming week.

@carlossanlop carlossanlop mentioned this issue Jun 23, 2021
Closed
@jkotas
Copy link
Member

jkotas commented Jun 24, 2021 •
edited
Loading

The last error returns Bad file descriptor. Descriptor can be bad when it's closed. But how is it possible that it got closed since it has not been shared with anyone yet? (line 74)

The easiest explanation is that some other thread closed the file descriptor in the meantime. File descriptors are small numbers, and get reused heavily. This is a potential sequence of events:

Thread A: calls close
Thread B: calls open - returns same fd as Thread A just closed
Thread A: calls close second time (bug!). It closes fd that Thread B is holding to now.
Thread B: calls lstat and gets EBADF

This failure pattern is consistent with what happened in #54615. The code called pipe syscall and immediately got EBADF trying to operate on the pipe file descriptors.

So I would be looking for a buggy code that is calling close twice.

@jkotas jkotas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jun 24, 2021
@danmoseley
Copy link
Member Author

danmoseley commented Jun 24, 2021 •
edited
Loading

Oh of course. Hmm, interesting, as just about everything uses SafeXHandle, so it should be hard to call close() twice.

More info from Kusto: first hit is 2021-06-15 21:36:40.9310000 (I assume it's UTC) after which it happens fairly often so I'd guess that's within a few hours of the problem being introduced. 9d771a2 went in at Tue Jun 15 20:59:58 2021 +0200.

It happens in both release and debug. It only happens when running System.IO.Tests.* - which is consistent with the bug being in the tests, but not conclusive. It hits mostly tests that enumerate files, perhaps because there's a lot of fd's involved. The biggest hitter is System.IO.Tests.DirectoryInfo_EnumerateFiles_str_str but there are many others.

It's evenly distributed across distros and macOS, but of course never Windows. Both x64 and Arm.

@danmoseley
Copy link
Member Author

Is this problematic?

runtime/src/libraries/System.IO.FileSystem/tests/RandomAccess/Base.cs

Lines 44 to 53 in 9d771a2

[Fact]
[SkipOnPlatform(TestPlatforms.Browser, "System.IO.Pipes aren't supported on browser")]
public void ThrowsNotSupportedExceptionForUnseekableFile()
{
using (var server = new AnonymousPipeServerStream(PipeDirection.Out))
using (SafeFileHandle handle = new SafeFileHandle(server.SafePipeHandle.DangerousGetHandle(), true))
{
Assert.Throws<NotSupportedException>(() => MethodUnderTest(handle, Array.Empty<byte>(), 0));
}
}

The SafeFileHandle takes ownership of the raw pipe handle. it is disposed and closes it, then the pipe is disposed and closes it again. This was introduce by above change.

danmoseley added a commit to danmoseley/runtime that referenced this issue Jun 24, 2021
@danmoseley
Fix sporadic double fd close
14b2cfb 
Fix dotnet#54589
@danmoseley danmoseley mentioned this issue Jun 24, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 24, 2021
@danmoseley danmoseley mentioned this issue Jun 24, 2021
Closed
stephentoub pushed a commit that referenced this issue Jun 24, 2021
@danmoseley
Fix sporadic double fd close (#54660)
f55390a 
Fix #54589
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 24, 2021
@adamsitnik adamsitnik mentioned this issue Jun 24, 2021
Merged
@adamsitnik
Copy link
Member

@danmoseley @tmds @jkotas big thanks for your help!!

With #54660 I can't reproduce it anymore.

We should call GetLastErrorInfo before calling Dispose, otherwise Dispose might change the error.

I've opened #54667 to address that

danmoseley added a commit that referenced this issue Jun 25, 2021
@danmoseley
Fix sporadic double fd close (#54660)
6deeab0 
Fix #54589
@ghost ghost locked as resolved and limited conversation to collaborators Jul 24, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@adamsitnik adamsitnik

Labels
area-System.IO blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' bug
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Fix sporadic double fd close danmoseley/runtime
5 participants
@tmds @stephentoub @adamsitnik @danmoseley @jkotas