bug: connection pooling on http/2 breaks with high concurrency

[lucacasonato]

I have an application that uses reqwest to make a large amount of outbound HTTP requests in a short timespan. It uses a single process wide reqwest::Client. It uses rustls for tls. and runs in a single threaded tokio event loop (the application is Deno).

When sending a large amount of requests concurrently, reqwest will reliably fail one of the requests with this error:

error sending request for url (https://deno-website2.now.sh/): http2 error: protocol error: not a result of an error

Full error

reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("deno-website2.now.sh")), port: None, path: "/", query: None, fragment: None }, source: hyper::Error(Http2, Error { kind: Proto(NO_ERROR) }) }

A reliable reproduction of this is to start 100 futures, each doing 20 requests to https://deno-website2.now.sh/ in series, and then polling them concurrently using a FuturesUnordered.

This is not an issue with the upstream server, as the exact same scenario (100 concurrent tasks, with 20 serial requests each) works fine with the Go std library with GOMAXPROCS=1.

You can view / try the reproduction here: https://github.com/lucacasonato/reqwest_connection_pooling_bug. Instructions for running are in the README.

It seems that this issue is related to connection pooling as using a single reqwest::Client per concurrent future does not exhibit the same behaviour. You can try this configuration by running the reproduction with --features no-pool.

[lucacasonato]

Confirmed with @piscisaureus that this is very likely the same root cause as hyperium/hyper#2500