Make the job retry delay configurable

[jdG-]

Is your feature request related to a problem? Please describe.
I have to call an API endpoint which is sometimes down for a few minutes/hours. So I want to retry exponentially instead after every failure.

Describe the solution you'd like
Simply being able to configure the retry strategy (pass the next settledAt to the fail method or something like that)

[michaelbromley]

I'm experimenting with a design like this:

const config = {
  plugins: [
    DefaultJobQueuePlugin.init({
      backoffStrategy: (queueName: string, attemptsMade: number) => {

        if (queueName === MY_FLAKY_API_QUEUE) {
          // An exponential backoff for the flaky API
          return Math.round((Math.pow(2, attemptsMade) - 1) * 100);
        }

        // This is the same as the current behaviour - instant retry.
        return 0;
      }
    }),
  ],
}; 

This allows you to configure backoffs for various queues in a central location.

This implementation extends the PollingJobQueueStrategy which is used in the SqlJobQueueStrategy (used by the DefaultJobQueuePlugin) and also in the InMemoryJobQueueStrategy.

I cannot build-in backoff logic into the JobQueueStrategy itself, since other strategies will implement their own entirely different retry backoff mechanism (e.g. Google Pub Sub has its own config).