HTTP2: REFUSED_STREAM not placed in graceful drain state?

[JamesNK]

Issue found during HTTP2 stress testing. One HttpClient is sending multiple requests in parallel, then canceling them via RST_STREAM sent to the server. For an unknown reason (#13406) the server and client disagree on the active stream count, and the sever eventually sends RST_STREAM (REFUSED_STREAM) and GOAWAY (STREAM_CLOSED).

Stream closed with REFUSED_STREAM should go into a graceful drain state and additional content sent to it from the client is ignored.

@Tratcher:

Look back in the trace to frame 281508. A bunch of server frames were bundled together, including a RST_STREAM (REFUSED_STREAM) and GOAWAY (STREAM_CLOSED).

It looks like stream 206249 was a POST immediately followed by data. That stream gets refused for some reason we’ll need to investigate, there should be something in the server logs.

Because the stream was refused, it was never added to the stream collection and placed into a graceful drain state. When the data frame is parsed it’s treated as a protocol error (frame sent to a closed stream) and the server initiates a GOAWAY.

Wireshark log: https://www.dropbox.com/s/5ndspth4t0qjyj6/clientreset-error.zip?dl=0

Exception on the client:

Unhandled exception. System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
   at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
   at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Grpc.Net.Client.Internal.GrpcCall`2.SendAsync(HttpClient client, HttpRequestMessage message)
   at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersAsync()

Repo is currently not public. Contact @JamesNK or @Tratcher for the repo source code and instruction.

[Tratcher]

Note this issue is secondary to figuring out why the stream was refused in the first place (#13406). It seems the client and server still disagree on the number of active streams. The scenario for this repro involved frequent cancellations/RSTs by the client for POST requests.

[JamesNK]

This is a bug but not a critical one. 3.1 fix?

[analogrelay]

We can see about this for 3.1 but the bar is pretty high. I'd expect this to bump.

[Tratcher]

#17484 looks like another scenario that can trigger this: Start a connection and then flood it with streams before you get the first Settings that tell you the actual stream limit. The streams are refused OK, but any subsequent data frames cause the connection to fault.
Http2ConnectionErrorException: HTTP/2 connection error (STREAM_CLOSED): The client sent a DATA frame to closed stream ID 9.

[analogrelay]

Going to dupe this to the other issue since the other issue has a concrete scenario ;).