DualModeAcceptAsync.AcceptAsyncV*BountTo* tests failed with TimeoutException

[ghost]

Tests:

• System.Net.Sockets.Tests.DualModeAcceptAsync.AcceptAsyncV6BoundToAnyV6_Success (any platform test)
• System.Net.Sockets.Tests.DualModeAcceptAsync.AcceptAsyncV4BoundToSpecificV6_CantConnect (Windows-only test)
• System.Net.Sockets.Tests.DualModeAcceptAsync.AcceptAsyncV6BoundToSpecificV4_CantConnect (Windows-only test)

Opened on behalf of @JiayiLi1

System.TimeoutException : Timed out while waiting for either of client and server connections...
Stack Trace:
at System.Net.Sockets.Tests.DualModeAcceptAsync.DualModeConnect_AcceptAsync_Helper(IPAddress listenOn, IPAddress connectTo) in /Users/buildagent/agent/_work/28/s/corefx/src/System.Net.Sockets/tests/FunctionalTests/DualModeSocketTest.cs:line 1016

Build : 2.0.0 - 20170704.01 (Core Tests)
Failing configurations:
- OSX.1012.Amd64-x64
- Release

Detail: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170704.01/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.DualModeAcceptAsync~2FAcceptAsyncV6BoundToAnyV6_Success

[karelz]

One occurrence only, moving to 2.1

[KristinXie1]

failed again in master, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170823.04/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.DualModeAcceptAsync~2FAcceptAsyncV6BoundToSpecificV4_CantConnect

The test System.Net.Sockets.Tests.DualModeAcceptAsync/AcceptAsyncV6BoundToSpecificV4_CantConnect has failed.

Assert.Throws() Failure\r
Expected: typeof(System.Net.Sockets.SocketException)\r
Actual: typeof(System.TimeoutException): Timed out while waiting for either of client and server connections...

    Stack Trace:

       at System.Net.Sockets.Tests.DualModeAcceptAsync.DualModeConnect_AcceptAsync_Helper(IPAddress listenOn, IPAddress connectTo)
   at System.Net.Sockets.Tests.DualModeAcceptAsync.<AcceptAsyncV6BoundToSpecificV4_CantConnect>b__4_0()

Build : Master - 20170823.04 (Core Tests)
Failing configurations:

• Windows.10.Amd64-x86
  • Release

[Sunny-pu]

failed again in CI:
https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_windows_nt_debug/lastCompletedBuild/testReport/System.Net.Sockets.Tests/DualModeAcceptAsync/AcceptAsyncV4BoundToSpecificV4_Success/

[karelz]

Helix official runs history shows only 1 failure of AcceptAsyncV6BoundToSpecificV4_CantConnect test and no failures for AcceptAsyncV6BoundToAnyV6_Success test.
Closing. Please reopen if it happens again.

[AriNuer]

Test System.Net.Sockets.Tests.DualModeAcceptAsync.AcceptAsyncV6BoundToSpecificV4_CantConnect failed with similar exception:

Error message
Assert.Throws() Failure
Expected: typeof(System.Net.Sockets.SocketException)
Actual: typeof(System.TimeoutException): Timed out while waiting for either of client and server connections...
---- System.TimeoutException : Timed out while waiting for either of client and server connections...

Stack trace
   at System.Net.Sockets.Tests.DualModeAcceptAsync.DualModeConnect_AcceptAsync_Helper(IPAddress listenOn, IPAddress connectTo) in /_/src/System.Net.Sockets/tests/FunctionalTests/DualModeSocketTest.cs:line 973
   at System.Net.Sockets.Tests.DualModeAcceptAsync.<AcceptAsyncV6BoundToSpecificV4_CantConnect>b__4_0() in /_/src/System.Net.Sockets/tests/FunctionalTests/DualModeSocketTest.cs:line 914
----- Inner Stack Trace -----
   at System.Net.Sockets.Tests.DualModeAcceptAsync.DualModeConnect_AcceptAsync_Helper(IPAddress listenOn, IPAddress connectTo) in /_/src/System.Net.Sockets/tests/FunctionalTests/DualModeSocketTest.cs:line 973
   at System.Net.Sockets.Tests.DualModeAcceptAsync.<AcceptAsyncV6BoundToSpecificV4_CantConnect>b__4_0() in /_/src/System.Net.Sockets/tests/FunctionalTests/DualModeSocketTest.cs:line 914

Details:
https://dnceng.visualstudio.com/public/_build/results?buildId=329154&view=ms.vss-test-web.build-test-results-tab&runId=9625064&resultId=170822&paneView=debug

[ViktorHofer]

@AriNuer if you find new occurrences of an existing failure please either reopen the issue or create a new one.

System.Net.Sockets.Tests.DualModeAcceptAsync.AcceptAsyncV6BoundToAnyV6_Success failed again: https://dnceng.visualstudio.com/public/_build/results?buildId=329154&view=ms.vss-test-web.build-test-results-tab&runId=9625064&resultId=170821&paneView=debug

[wfurt]

I see 19 failures in last 90 days. All on flavors of Windows 10. The artifacts are unfortunately gone. On next failure we should collect console.log and testResult.xml as they may contain useful information.
We should also instrument ConnectClient to log any exceptions to output log stream. (perhaps other places as well).
We should also dump state of client and server in case of timeout.

[ViktorHofer]

Happened again: https://dnceng.visualstudio.com/public/_build/results?buildId=366911&view=ms.vss-test-web.build-test-results-tab&runId=11102492&paneView=debug&resultId=178926

console.cf0a6305.log
testResults (1).zip

[safern]

Happened again: https://dev.azure.com/dnceng/public/_build/results?buildId=375425&view=ms.vss-test-web.build-test-results-tab&runId=11499012&resultId=163584&paneView=debug

[karelz]

Failures during 3/12-8/24 (incl. PRs):

Date	Build / PR	OS	Test
4/1	PR #50239	Windows.10.Amd64.Server19H1.ES.Open	AcceptAsyncV4BoundToSpecificV6_CantConnect
5/29	PR #53467	Windows.10.Amd64.Server19H1.ES.Open - x86	AcceptAsyncV6BoundToAnyV6_Success
7/27	PR #56294	Windows.10.Amd64.Server19H1.Open - x86	AcceptAsyncV4BoundToSpecificV6_CantConnect
8/20	PR #57833 - release/6.0 branch	Windows.10.Amd64.Server19H1.ES.Open	AcceptAsyncV6BoundToAnyV6_Success

Kusto query:

let failedTests = (methodName : string, includePR : bool, messageSubstr: string) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').TestResults
| where Type startswith "System.Net" 
    and Result == 'Fail'
| where Method == 'AcceptAsyncV6BoundToAnyV6_Success' or Method == 'AcceptAsyncV4BoundToSpecificV6_CantConnect' or Method == 'AcceptAsyncV6BoundToSpecificV4_CantConnect'
| where Message contains messageSubstr
| distinct JobId, WorkItemId, Message, Exception, StackTrace, Method, Type //, Arguments
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
    | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
    | where Type startswith "test/functional/cli/"
        and not(Properties contains "runtime-staging")
    | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
};
failedTests('', true, '');

[wfurt]

triage: dup of #1481. We should make dual mode more reliable. No recent failures.