Hanging/Freezing persistent subscription test on net48 on CI

[shaan1337]

The test EventStore.Client.Bugs.Issue_1125.persistent_subscription_delivers_all_events hangs on CI on the net48 build in this PR: #93

It was thus skipped temporarily in this commit until we figure out the root cause: 92804a7

[shaan1337]

Progress updates:

• Managed to consistently reproduce the issue on Ubuntu 18.04 by adding GRPC_TRACE=all and GRPC_VERBOSITY=debug to the environment which slows down the execution and makes the issue much more likely to occur. To analyze the grpc traces, I also had to redirect standard output and standard error to a file with a dup2 system call but not sure it's a necessary condition to make the issue reproducible.

• Only a single iteration of the test is required to reproduce the issue (instead of 50 in the test)

• Analyzing the backtraces with lldb shows that the mono Finalizer thread is waiting on a futex. I'm not 100% sure this is the cause but it's very likely given the information below - the Dispose calls are probably being done by the Finalizer thread. The fact that the test successfully passes and then the hanging occurs also supports this hypothesis.

• The issue occurs when we call Dispose() on AsyncDuplexStreamingCall - it seems that it's currently in use only with persistent subscriptions.

• Removing _call?.Dispose(); here stops the hanging:

  EventStore-Client-Dotnet/src/EventStore.Client.PersistentSubscriptions/PersistentSubscription.cs

  Line 206 in 0cefa70

  _call?.Dispose();

• It is also related to the dispose calls on the AsyncDuplexStreamingCall in the interceptors - replacing any of the response.Dispose calls with an empty action stops the hanging:
  

  EventStore-Client-Dotnet/src/EventStore.Client/Interceptors/ReportLeaderInterceptor.cs

  Line 48 in 06aad41

  response.GetStatus, response.GetTrailers, response.Dispose);

  
  

  EventStore-Client-Dotnet/src/EventStore.Client/Interceptors/TypedExceptionInterceptor.cs

  Line 93 in 06aad41

  response.Dispose);

Summary:

It appears that multiple simultaneous calls to Dispose() on AsyncDuplexStreamingCall causes a deadlock due to a race condition.

I've attempted a minimal reproduction (without EventStore-specific code) but I've been unsuccessful so far.

[shaan1337]

Managed to get a minimal repro of the issue: https://github.com/shaan1337/async-duplex-streaming-call-hang
Opened an issue on gRPC repo: grpc/grpc#25488

[tambeau]

PR: #93 is closed

[timothycoleman]

In the end dealt with in #172 by dropping the (short lived) support for net48