Option to ignore hubcaps failures

[DanySK]

Hi,
I'm using lychee to check link integrity for a whole website.
Does the tool check the same link multiple times? Despite providing a valid token, I get many:

✗ http://github.com/<foo>: Network error (hubcaps)

That I would not expect.

Is there a way to tell lychee to ignore these errors?
Something like --ignore-error=429 would be great.

Alternatively or in addition, would it be possible to wait and retry after a 429?
Something like --wait-after-cap=60 to retry links after a minute if the cap has been reached.

[mre]

You can use

lychee --accept 200,429 ...

We should make the retry interval configurable. It's currently making three attempts and waits 1, 2, and 4 seconds respectively between each retry.
The goal is to use tower to write a service around the client in the long run and properly handle rate limiting and backoff there.

• 429 Too Many Requests #367
• Look into tower as a replacement for deadpool + channels #36

[mre]

@DanySK did this fix your issue?

[DanySK]

Partly: it can skip 429 entirely, but retry count and timeout can't be set.

[mre]

True, we should make those configurable. I just keep forgetting to do it. 😅

[mre]

The retry count is already configurable through --max-retries, the timeout can be configured with --timeout and for the wait time between requests I created a pull request at #525.

[mre]

--retry-wait-time is in master now. Let me know if that solves your problem @DanySK. 😊

[mre]

A proper solution would be #36, with which we track the individual state of every website. This way we avoid overloading the servers and can do proper rate-limiting by parsing the respective headers and storing them in the website state.
For now, increasing the --retry-wait-time should at least help in simple cases.

[DanySK]

I'm waiting for the option to hit docker hub, as soon as it does I'll try it out

[mre]

Hum... it should already be in.
Did you try

docker pull lycheeverse/lychee:master

I'm away from my computer so I can't verify.

[DanySK]

I am using latest, which I guess points to the latest stable then?

[mre]

Yes, latest gets updated on new releases only, not on merges to master.

[mre]

Guess we can close this. latest should have the changes and the next release of lychee will as well. Using tower for retries is tracked in #36.