Asynchronous processing of redirects in request_multiple() when using curl transport

[pprkut]

Is your feature request related to a problem?

I think this is one of those edge cases where it could be interpreted as either a bug or a missing feature. What's happening is this:

When calling Requests::request_multiple() with multiple requests, curl_multi_exec() will process them asynchronously in the background. Requests then loops over the CurlMultiHandle to handle the requests that finished, potentially out of order. So far so good.

In the Curl transport, when processing the finished request, it calls the transport.internal.parse_response hook, which the Requests class hooks into in https://github.com/WordPress/Requests/blob/develop/src/Requests.php#L531 or https://github.com/WordPress/Requests/blob/develop/src/Requests.php#L531. This leads to Requests::parse_response() call, which also handles redirects by calling Requests::request().

The effect of this is that currently redirects are processed synchronously when encountered after every asynchronous request, which in a worst case scenario, where every request passed to Requests::request_multiple() leads to a redirect and $options['follow_redirects'] is true, essentially causes all requests to be processed near synchronously.

Describe the solution you'd like

I'd think the expectation of calling Requests::request_multiple() is that the requests are handled asynchronously, including potential redirects. What I don't know is the best way to achieve that.
I suppose the most ideal would be if any redirect adds a handle to the active CurlMultiHandle so we can continue processing them in the existing loop. I'm not sure that's possible, however (quite likely it's not). So the next best thing would be a new, separate call to curl_multi_exec(), which at least causes the redirect to be put in the background so we could choose to process the other responses first before coming back to this one.
The challenge here is that this too should then trigger all the usual hooks (requests.before_redirect_check, requests.before_redirect).

Describe alternatives you've considered

Another option would be to handle redirects in the Requests class by collecting all responses that indicate a redirect and issuing a new call to Curl::request_multiple() for those. This would be easier to do I suppose, at the cost of less concurrency.

• I intend to create a pull request to implement this feature myself.

I can try at least 🙂