Server restart exhausts retries

[jkozlowski]

What happened?

Tests in palantir/atlasdb#5233 would fail because of NoHttpResponseException. The nature of this test is that it gracefully shuts down servers to test resiliency to that scenario. The dialogue clients are setup with a single URL only, meaning that all previously open connections are closed, and the request fails because it blows through it's retry limit.

What did you want to happen?

The request should succeed: we know the server is up because the tests await for that.

[jkozlowski]

Idea: if we get a NoHttpResponseException, and it took less than 100ms, we should retry with another connection. Max to 100 times? Up to 1 sec?

This would basically live at the per-host channel level, we'd need a retry loop there, that's different to the RetryingChannel.

[Regan-Koopmans]

@jkozlowski I wonder whether we could use exponential backoff to increase the retry delay?

[jkozlowski]

We already have exponential backoff in RetryingChannel; the problem here is that all the connections are blown, because the server restarted. But we cannot know that without doing a read sadly.

I think this isn't much of a problem in prod, because you'll most likely land on another host, but in this test it fails.