Take steps to mitigate cascading failure

[orangejulius]

This package instructs superagent to perform retries of HTTP requests that fail. This is good for fixing issues such as random network glitches, etc. However, if not done carefully it can cause chain reactions that completely break systems operating close to the edge of full capacity.

We should ensure superagent backs off exponentially, with some random jitter added, between request retries.

Without that, a heavily loaded server that does not respond within the timeout will be bombarded with more and more requests in the form of retries. This further increases the load and reduces its capacity to return correct results.

The jitter is necessary because its possible for retry requests to "sync up" through various means, which can cause a normally tolerable load on a downstream server to become too much to handle.

This may require either forking superagent or going back to our own retry code.

For more info see the excellent Addressing Cascading Failures section of the Google SRE book.

[missinglink]

There is probably already a module which does what you're looking for.

What about this one? https://www.npmjs.com/package/superagent-retry-delay