Custom conditions for retrying proxy requests

[mholt]

As discussed in #4245 (comment), it could be useful to have a way to further customize when requests are retried by the load balancer while proxying. As far as I know, all of that comment is now supported/implemented except this part:

Maybe it should be possible to retry for some of these cases, on an opt-in basis (taken from https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#4.2-retry-on). Some of these can be risky but we should let the user choose if they want it:

• Empty response (no response body) or socket closed before receiving a response
• On TLS handshake error (I'm not sure if we already do for these or not, but writing it down here for now)
• Response timeout (i.e. response_header_timeout HTTP transport option, or read_timeout fastcgi transport option), i.e. server took too long to respond
• By response status code (e.g. server actually responded with 503 Service Unavailable from an upstream proxy)

As always, non-GET/HEAD/OPTIONS requests that are retried for any reason other than "the connection to the backend failed" is unsafe by default, so the user would have to carefully configure conditions in which the request is idempotent and may be retried. One common example is an Idempotency-Key header which the backend app must honor (properly implement).

Opening this issue as a more focused discussion on just this remaining facet of the whole previous discussion.

[jopaleti]

@mholt I will like to work on this if I can be assigned

[mholt]

Sure, feel free to reply with your proposal or submit a PR (expect it to be iterated upon). Thanks! 👍

[jopaleti]

@mholt Alright I am working on it