ASP.NET core GRPC server can terminate connections when MAX_CONCURRENT_STREAMS is breached

[djluck]

What version of gRPC and what language are you using?

grpc=v2.24.0
lang=C#

What operating system (Linux, Windows,...) and version?

Windows, v10.0.18362 Build 18362

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

v3.0.100

What did you do?

When issuing many requests on a fresh/ idle connection, it's possible to exceed the MAX_CONCURRENT_STREAMS setting of a server, causing the connection to fault and failing successfully processed streams.
Exceeding the MAX_CONCURRENT_STREAMS on a fresh/ idle connection should be supported as the initial value is assumed to be unlimited until a peer has communicated it's prefered limit.

This psuedo-code highlights how to re-create the issue on the client-side:

// This test fails about 1/3 times (yuk). Bit hacky but let's repeat the test a large number of times
// to verify everything is working correctly.
for (int i = 0; i < 5; i++)
{
	// Must use a fresh client for each iteration of the test. The GRPC server will indicate it's max concurrency
	// limit after the first request successfully completes. However, we are interested in making sure large bursts
	// on a connection that has not discovered this limit works as expected.
	using var c = GrpcChannel.ForAddress(new Uri("https://localhost:5001"));
	var client = new Health.HealthClient(c);
	
	var tasks = Enumerable.Range(1, 5)
		.Select(x => Task.Run(async () => await client.CheckAsync(new HealthCheckRequest())))
		.ToArray();

	await Task.WhenAll(tasks);
}

MAX_CONCURRENT_STREAMS should be set to a low value in the ASP.NET grpc server (in appsettings.json):

{
  "Kestrel": {
    "Limits": {
      "Http2" : {
        "MaxStreamsPerConnection" : 2
      }
    }
  }
}

What did you expect to see?

The streams that exceed the MAX_CONCURRENT_STREAMS limit should be closed and any DATA frames received by the server for these streams should result in a STREAM_CLOSED error as per the HTTP/2 RFC:

What did you see instead?

Streams that exceed the MAX_CONCURRENT_STREAMS had DATA frames in-flight which resulted in the entire TCP connection being torn down:

[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received SETTINGS frame for stream ID 0 with length 12 and flags NONE
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received HEADERS frame for stream ID 1 with length 139 and flags END_HEADERS
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received HEADERS frame for stream ID 3 with length 139 and flags END_HEADERS
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received HEADERS frame for stream ID 5 with length 139 and flags END_HEADERS
[10:13:55 DBG] Connection id "0HLRKB2HNVILT": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 5 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
[10:13:55 INF] Request starting HTTP/2 POST https://localhost:5001/grpc.health.v1.Health/Check application/grpc
[10:13:55 INF] Request starting HTTP/2 POST https://localhost:5001/grpc.health.v1.Health/Check application/grpc
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" sending RST_STREAM frame for stream ID 5 with length 4 and flags 0x0
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received HEADERS frame for stream ID 7 with length 139 and flags END_HEADERS
[10:13:55 DBG] Connection id "0HLRKB2HNVILT": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 7 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" sending RST_STREAM frame for stream ID 7 with length 4 and flags 0x0
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received HEADERS frame for stream ID 9 with length 139 and flags END_HEADERS
[10:13:55 DBG] Connection id "0HLRKB2HNVILT": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 9 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" sending RST_STREAM frame for stream ID 9 with length 4 and flags 0x0
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received SETTINGS frame for stream ID 0 with length 0 and flags ACK
[10:13:55 VRB] Connection id "0HLRKB2HNVILT" received DATA frame for stream ID 9 with length 5 and flags NONE
[10:13:55 DBG] Connection id "0HLRKB2HNVILT": HTTP/2 connection error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2ConnectionErrorException: HTTP/2 connection error (STREAM_CLOSED): The client sent a DATA frame to closed stream ID 9.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessDataFrameAsync(ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
[10:13:55 DBG] Connection id "0HLRKB2HNVILT" is closed. The last processed stream ID was 9.

The impact of faulting the entire connection is that some requests may have successfully processed and their CancellationToken may not trigger- this means it's impossible to recover from this kind of error gracefully.

Anything else we should know about your project / environment?

[JamesNK]

Thanks for the detailed repro. This looks like a Kestrel HTTP/2 issue rather than something in gRPC. I've copied it here - dotnet/aspnetcore#17484

[djluck]

Thanks for moving the issue to its proper home, appreciate it!