Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Threading.Tests.MutexTests tests failing on Linux #28449

Closed
ahsonkhan opened this issue Jan 18, 2019 · 3 comments · Fixed by #36268
Closed

System.Threading.Tests.MutexTests tests failing on Linux #28449

ahsonkhan opened this issue Jan 18, 2019 · 3 comments · Fixed by #36268
Labels
area-System.Threading disabled-test The test is disabled in source code against the issue test-bug Problem in test source code (most likely)
Milestone

Comments

@ahsonkhan
Copy link
Member

From dotnet/corefx#34500

Debian.8.Amd64.Open-x64-Release
https://mc.dot.net/#/user/tkp1n/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/ecffe5046608086201c8716ae132ddacbd7182d8/workItem/System.Threading.Tests/analysis/xunit/System.Threading.Tests.MutexTests~2FCrossProcess_NamedMutex_ProtectedFileAccessAtomic(prefix:%20%5C%225e65042415ce448f872dae5fb40e7a74%5C%22)

System.Threading.Tests.MutexTests/CrossProcess_NamedMutex_ProtectedFileAccessAtomic(prefix: "5e65042415ce448f872dae5fb40e7a74")

Message :
Assert.True() Failure
Expected: True
Actual:   False
Stack Trace :
   at System.Threading.Tests.ThreadTestHelpers.<>c__DisplayClass4_0.<CreateGuardedThread>b__2() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 56
   at System.Threading.Tests.ThreadTestHelpers.RunTestInBackgroundThread(Action test) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 108
   at System.Threading.Tests.MutexTests.CrossProcess_NamedMutex_ProtectedFileAccessAtomic(String prefix) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Threading/tests/MutexTests.cs:line 262

Debian.8.Amd64.Open-x64-Release
https://mc.dot.net/#/user/tkp1n/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/ecffe5046608086201c8716ae132ddacbd7182d8/workItem/System.Threading.Tests/analysis/xunit/System.Threading.Tests.MutexTests~2FCrossProcess_NamedMutex_ProtectedFileAccessAtomic(prefix:%20%5C%22Local%5C%5C%5C%5C65f91a48792b41ba9b7e699108f7de52%5C%22)

System.Threading.Tests.MutexTests/CrossProcess_NamedMutex_ProtectedFileAccessAtomic(prefix: "Local\\65f91a48792b41ba9b7e699108f7de52")

Message :
System.AggregateException : One or more errors occurred. (Assert.True() Failure
Expected: True
Actual:   False)
---- Assert.True() Failure
Expected: True
Actual:   False
Stack Trace :
   at System.Threading.Tests.ThreadTestHelpers.<>c__DisplayClass4_0.<CreateGuardedThread>b__1() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 51
   at System.Threading.Tests.ThreadTestHelpers.<>c__DisplayClass4_0.<CreateGuardedThread>b__2() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 57
   at System.Threading.Tests.ThreadTestHelpers.RunTestInBackgroundThread(Action test) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 108
   at System.Threading.Tests.MutexTests.CrossProcess_NamedMutex_ProtectedFileAccessAtomic(String prefix) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Threading/tests/MutexTests.cs:line 262
----- Inner Stack Trace -----
   at System.Threading.Tests.ThreadTestHelpers.CheckedWait(WaitHandle wh) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 133
   at System.Threading.Tests.MutexTests.IncrementValueInFileNTimes(Mutex mutex, String fileName, Int32 n) in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Threading/tests/MutexTests.cs:line 268
   at System.Threading.Tests.MutexTests.<>c__DisplayClass15_0.<CrossProcess_NamedMutex_ProtectedFileAccessAtomic>b__0() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/System.Threading/tests/MutexTests.cs:line 257
   at System.Threading.Tests.ThreadTestHelpers.<>c__DisplayClass4_0.<CreateGuardedThread>b__0() in /mnt/j/workspace/dotnet_corefx/master/linux-TGroup_netcoreapp+CGroup_Release+AGroup_x64+TestOuter_false_prtest/src/Common/tests/System/Threading/ThreadTestHelpers.cs:line 35

cc @kouvel

@stephentoub
Copy link
Member

stephentoub commented Jul 29, 2019

Again:
https://dev.azure.com/dnceng/public/_build/results?buildId=283773&view=ms.vss-test-web.build-test-results-tab&runId=7926858&paneView=attachments&resultId=393950

    System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.PostAsyncExpect100Continue_NonSuccessResponse_RequestBodyNotSent [FAIL]
      Assert.Null() Failure
      Expected: (null)
      Actual:   Byte[] [42, 42, 42, 42, 42, ...]
      Stack Trace:
        /_/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs(1926,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.<>c__DisplayClass67_0.<<PostAsyncExpect100Continue_NonSuccessResponse_RequestBodyNotSent>b__1>d.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
        /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(71,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/Common/tests/System/Net/Http/Http2LoopbackServer.cs(186,0): at System.Net.Test.Common.Http2LoopbackServer.CreateClientAndServerAsync(Func`2 clientFunc, Func`2 serverFunc, Int32 timeout)
        /_/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Http2.cs(1898,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Http2.PostAsyncExpect100Continue_NonSuccessResponse_RequestBodyNotSent()
        --- End of stack trace from previous location where exception was thrown ---

@msftgits msftgits transferred this issue from dotnet/corefx Feb 1, 2020
@msftgits msftgits added this to the 5.0 milestone Feb 1, 2020
@maryamariyan maryamariyan added the untriaged New issue has not been triaged by the area owner label Feb 23, 2020
@stephentoub stephentoub removed the untriaged New issue has not been triaged by the area owner label Feb 25, 2020
@stephentoub stephentoub modified the milestones: 5.0, Future Feb 25, 2020
kouvel added a commit to kouvel/runtime that referenced this issue May 12, 2020
Below when I refer to "mutex" I'm referring to the underlying mutex object, not an instance of the `Mutex` class.
- When the last reference to a mutex is closed while the lock is held by some thread and a pthread mutex is used, the mutex was attempted to be destroyed but that has undefined behavior
- There doesn't seem to be a way to behave exactly like on Windows for this corner case, where the mutex is destroyed when the last reference to it is released, regardless of which process has the mutex locked and which process releases the last reference to it (they could be two different processes), including in cases of abrupt shutdown
- For this corner case I settled on what seems like a decent solution and compatible with older runtimes:
  - When a process releases its last reference to the mutex
    - If that mutex is locked by the same thread, the lock is abandoned and the process no longer references the mutex
    - If that mutex is locked by a different thread, the lifetime of the mutex is extended with an implicit ref. The implicit ref prevents this or other processes from attempting to destroy the mutex while it is locked. The implicit ref is removed in either of these cases:
      - The mutex gets another reference from within the same process
      - The thread that owns the lock exits and abandons the mutex, at which point that would be the last reference to the mutex and the process would not reference the mutex anymore
  - The implementation based on file locks is less restricted, but for consistency that implementation also follows the same behavior
- There was also a race between an exiting thread abandoning one of its locked named mutexes and another thread releasing the last reference to it, fixed by using the creation/deletion process lock to synchronize

Fix for dotnet#34271 in master
Closes dotnet#28449 - probably doesn't fix the issue, but trying to enable it to see if it continues to fail
kouvel added a commit that referenced this issue May 13, 2020
Fix Unix named mutex crash during some race conditions

Below when I refer to "mutex" I'm referring to the underlying mutex object, not an instance of the `Mutex` class.
- When the last reference to a mutex is closed while the lock is held by some thread and a pthread mutex is used, the mutex was attempted to be destroyed but that has undefined behavior
- There doesn't seem to be a way to behave exactly like on Windows for this corner case, where the mutex is destroyed when the last reference to it is released, regardless of which process has the mutex locked and which process releases the last reference to it (they could be two different processes), including in cases of abrupt shutdown
- For this corner case I settled on what seems like a decent solution and compatible with older runtimes:
  - When a process releases its last reference to the mutex
    - If that mutex is locked by the same thread, the lock is abandoned and the process no longer references the mutex
    - If that mutex is locked by a different thread, the lifetime of the mutex is extended with an implicit ref. The implicit ref prevents this or other processes from attempting to destroy the mutex while it is locked. The implicit ref is removed in either of these cases:
      - The mutex gets another reference from within the same process
      - The thread that owns the lock exits and abandons the mutex, at which point that would be the last reference to the mutex and the process would not reference the mutex anymore
  - The implementation based on file locks is less restricted, but for consistency that implementation also follows the same behavior
- There was also a race between an exiting thread abandoning one of its locked named mutexes and another thread releasing the last reference to it, fixed by using the creation/deletion process lock to synchronize

Fix for #34271 in master
Closes #28449 - probably doesn't fix the issue, but trying to enable it to see if it continues to fail
@ghost ghost locked as resolved and limited conversation to collaborators Dec 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Threading disabled-test The test is disabled in source code against the issue test-bug Problem in test source code (most likely)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants