Improved retry/connection handling of Apache HTTP Client

[FloEberle]

Hello everyone

While running long running tests with karate version v1.5.0 (latest as I'm writing this) in the CI of my company projects, we are experiencing flaky tests due to the underlying Apache HTTP client. I discovered the issue #2408 and the related configuration change and we adapted our configuration accordingly.
It turns out, this does not fully fix the issue in our case.

We observe randomly occurring errors of the following kind:

11:25:25.294 [pool-2-thread-2] ERROR com.intuit.karate - Thrown an exception Read timed out
11:25:25.297 [pool-2-thread-2] ERROR com.intuit.karate - java.net.SocketTimeoutException: Read timed out, http call failed after 5008 milliseconds for url: http://localhost:8115/api/path
11:25:25.297 [pool-2-thread-2] ERROR com.intuit.karate - classpath:karate/path/to/feature:11
When method post
http call failed after 5008 milliseconds for url: http://localhost:8115/api/path
java.net.SocketTimeoutException: Read timed out classpath:karate/path/to/feature:11

Increasing the timeout to astronomic values like 10 minutes did not resolve this and the api was responsive during the time the test failed.

The test then might be replayed and either randomly succeeds or fails in a random different feature, sadly I was not able to create an isolated MCVE from the issue.

I debugged this down to point where I suspect that Apache HTTP Client is using half open connection or operates on connection with unusable/closed sockets. Disclaimer: I would not consider myself an expert in this area!

I was able to resolve (or better: circumvent) this by either:
a) Adding a ConnectionPool in ApacheHttpClient L248 which validates inactive connections

        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
        connectionManager.setValidateAfterInactivity(500);
        clientBuilder.setConnectionManager(connectionManager);

OR b) catch and handle SocketTimeoutException similar to NoHttpResponseException in CustomHttpRequestRetryHandler and increase the executionCount

Im not sure If i should pursuit one of the approaches further and submit a PR and await your advise, but I'd definitely love to contribute here.

[ptrthomas]

@FloEberle thanks for the research. it has been a while, but can you look at #2487 and see if that approach solves the problem. this does depend on upgrading to apache http client v5 #2471 which I now consider a priority

[FloEberle]

@ptrthomas Thanks for pointing this out. I will implement a temporary workaround of automated retrying on our end until those PRs are released and subscribed to them to stay in the loop.
I leave the decision to you if you want to close this issue or not.