Retry count for validate links task #490
Conversation
| def validateWithRetries(retryCount: Int): Connection.Response = { | ||
| val res = tryConnect | ||
| if (retryCount == 0 || res.statusCode() == 200) res | ||
| else validateWithRetries(retryCount - 1) |
There was a problem hiding this comment.
Will it likely be the status codes that signals temporary unavailable rather than not being able to connect for example (so perhaps throws and is caught further down rather than retried right now)?
Also - should it be more selective with what error codes it retries for (at least some are permanent or likely permanent 400-407 for example), and perhaps back off a little since immediate retry if the server is overloaded will likely fail again.
There was a problem hiding this comment.
That make sense. I was specifically targeting use case where site is slow and jsoup times out. Was hoping at least one of the retries will succeed. Was planning to keep it simple unless people file issues/enhancements 😉
I ll look into status codes.
Do you think fist class backoff is necessary? I can definitely try simplest backoff solution.
Have not looked at code completely, but does link check happen in parallel? If not, backoff will introduce lot of delay.
There was a problem hiding this comment.
So if it is mostly about timeouts, would jsoup still return a response with a status code rather than throw an exception?
I don't think backoff is necessary.
There was a problem hiding this comment.
Yes, you are right, it should throw exception if endpoint does not return any response.
In my case, some sites were returning 503 for very short period of time.
There was a problem hiding this comment.
What do you think about explicitly checking for 503 (and perhaps 502, 503 as well, maybe 500) + connection timeout throw and retry only for those rather than across all other (less likely to be temporary) status codes.
There was a problem hiding this comment.
Yes, I am planning to submit that over the weekend. (Maybe earlier If i get the opportunity)
There was a problem hiding this comment.
I am retrying following status codes: 500, 502, 503, 504 (included additional 504 - Gateway Timeout) and SocketTimeoutException
core/src/main/scala/com/lightbend/paradox/ParadoxProcessor.scala
Outdated
Show resolved
Hide resolved
Fix #488