Add backoff option to pagination

[jaulz]

This PR adds an easy possibility to respect rate limits that are enforced by some APIs. If the rate limit is 600 per hour and you know that you will most likely hit the limit you can set backoff to 100 which means that paginated requests are triggered only every 100 milliseconds. Unfortunately, I had no clever idea how to implement a test but maybe you have an idea?

Checklist

• I have read the documentation.
• I have included a pull request description of my changes.
• I have included some tests.
• If it's a new feature, I have included documentation updates.

[szmarczak]

This needs tests

[jaulz]

Yep, definitely. That why I mentioned in the description that I had no clever idea how to test it 😉 Any idea how we could simulate this?

[szmarczak]

Faketimers (lolex)

[jaulz]

@szmarczak yeah, I saw that you used it in other tests as well but I couldn't really figure out how to properly use it in this case.

[szmarczak]

This PR adds an easy possibility to respect rate limits that are enforced by some APIs.

Actually do we need a backoff option? got.paginate() uses the Promise API under the hood, so it has got a retry mechanism. It also respects the Retry-After header.

[jaulz]

Retry means that we accept running into potential server errors whereas backoff would allow us to prevent server errors from the beginning if you know the rate limits.

[jaulz]

I was just about to give the tests another try but after installing all dependencies and run yarn testI see these issues:

Error: Failed to load config "xo-typescript" to extend from.
Referenced from: BaseConfig

Is there anything wrong on master at the moment?

[szmarczak]

Looks like you're missing npm install...

[jaulz]

Not really I think... I even removed the node_modules and did a fresh yarn.

[szmarczak]

What about if you use npm instead? We don't use yarn [yet?] :)

[jaulz]

Hm, honestly I have no clue how that test system works... I would have assumed that it would work with my current approach but then again I am running into typing issues (i.e. end does not exist on t).

[szmarczak]

No worries, I'll fix it.

[jaulz]

@szmarczak thanks! I'm curious how you will do it 😊 I couldn't even see that my test was executed. Is there actually a simple way to run a single test (if I use only I see lint issues) in watch mode? And is it also somehow possible to log some stuff in between for debugging?

[sindresorhus]

@jaulz Bump :)

[jaulz]

@sindresorhus yeah, I would still love to see this as part of the package but unfortunately I was stuck with the test set up 😞 @szmarczak could you by any chance find some time and fix the test?

[szmarczak]

I'll do this today. I'm not home rn, will be in two/three hours.

[szmarczak]

After 30+ minutes of trying to introduce fake-timers, I'm giving up. It doesn't have clock.runUntilNextAsync which makes things extremely hard. I'll go with native setTimeout.

[jaulz]

@szmarczak thanks a lot!

[szmarczak]

No problem, sorry I hadn't got to this before.