if new jobs are added while a set of rate-limiter jobs are in delayed, are the new jobs ran _after_ the delayed set?

[twelve17]

I've ran into a situation in which newer jobs added to the queue (no priority set) are being processed before older jobs in the queue (e.g. older job timestamp), when a rate limiter is in place.

I've got it down to where I can sometimes reproduce this with some test queue code I wrote.

The basic premise of the test is: queue a large number of messages, with 30 seconds of sleep time between each batch. Each batch had some overlap in job IDs as the previous batch.

The queue is configured with this limiter:

limiter: {
    max: 2000,
    duration: 30000, // 30 seconds
  },

The messages are queued (by looping and calling .add to the queue, as we were not sure if the bulk add preserved order). The batches are as follows:

• batch 1: start job id: 1, count: 5000 // would create 5000 messages named job-1, job-2, etc
• batch 2: start job id: 4500, count: 5000
• batch 3: start job id: 9000, count: 5000

(Again, between the above batches, the producer code slept for 30 secs.)

The processor job code just inserts a record to a db table, which includes:

• batch number
• job.timestamp
• timestamp of when the insert happened (proxy for job completion)

When the 3 batches were done, I queried the DB, in job completion order, for a job with batch number N being processed before a job with batch number N-1.

I have been unable to narrow this down to a simpler test case, unfortunately.

[manast]

rate-limit has been rewritten and improved in BullMQ so most likely this issue is not there, if you can, try to upgrade.

[twelve17]

We are trying to upgrade but it is taking a bit of time, as the setup including rate limiting is quite different than bull, for NestJS. (The BullMQ docs point to the NestJS queue docs, which suggest using Bull.).

In any event, is this possibly expected behavior? I noticed this comment in the codebase, which might be related to the issue:

// (up to 4096 jobs per given timestamp or 4096 jobs apart per timestamp)
//
// WARNING: Jobs that are so far apart that they wrap around will cause FIFO to fail

Perhaps the README or documentation can be updated to include this limitation? (I would be happy to submit a PR.)