Support duration based retries

[jonsyu1]

What happened?

Retry behavior is exposed via backoff duration and number of retries. Clients can specify waiting for up to 300ms before the first retry, up to 600ms before the next retry and so on until they run out retries.

Clients should think about how long they're willing to try rather than how many times they are willing to try. For example, an interactive dashboards are willing to retry for up to 10 seconds whereas long-running pipelines are willing to retry up to 20 minutes. While we can approximate this behavior with the existing controls by tuning the backoff duration and retry count, we're tuning a distribution, not a specific duration. For example, the expected time spent retrying could be 20 minutes but if we're unlucky, the total time spent retrying could be less than 2 minutes. This leads to clients that are pathologically overly sensitive to 429s from the server.

What did you want to happen?

Support duration based retries where the client specifies how long they're willing to retry. This allows the client to express their retry contract in a consistent, easy to understand way. We still get all the benefits of employing jitter to stagger retries, but the implementation details are hidden away from the client. The client tries for 20 minutes and then gives up saying that it has tried for 20 minutes. This means a lot more than giving up saying that it has tried 10 times.