Place delayed jobs at the end of the waiting list

[kietheros]

Description

I have a queue with rate limiter

limiter: {
    max: 1,
    duration: 4000,
    groupKey: 'sourceId'
}

// add 20 jobs with sourceId 'Z'
for (let i = 0; i < 20; i++) {
    await queue.add({
        sourceId: 'Z'
    }, {
        jobId: `Z:${i+1}`
    });
}

// add 100 jobs with sourceId 'A', 'B', 'C', 'D', 'E'
for (let i = 0; i < 20; i++) {
    let sourceIds = ['A', 'B', 'C', 'D', 'E'];
    for (let j = 0; j < 5; j++) {
        const sourceId = sourceIds[j];
        await queue.add({
                sourceId: sourceId
            }, {
                jobId: `${sourceId}:${i+1}`,
            });
    }
}

queue.process(1, async (job) => {
    logger.debug('Start job', job.id);
    await sleep(2000); // sleep 2s
    logger.debug('Finish job', job.id);
});

Because of rate limiter, some jobs of source Z are moved into delayed queue. After amount of time, jobs of source Z are placed at the beginning of waiting list. The beginning of waiting list here is the left side, so 'Z' jobs will not be processed as soon as the worker is idle.

I have read docs

a delayed job will not be processed directly, instead it will be placed at the beginning of the waiting list and processed as soon as a worker is idle.

I wonder that these jobs have to be placed at the end of the waiting list (right side) to be processed as soon as a worker is idle. ???
Could anyone explain me about this problem ? Thanks !

Minimal, Working Test code to reproduce the issue.

(An easy to reproduce test case will dramatically decrease the resolution time.)

Bull version

3.27.0

Additional information

[kietheros]

@manast
Might I take a moment of your time to ask a small question !!!

In above test case, why delayed jobs are placed at the left of waiting list, and will not be processed immediately, because workers pop jobs from the right of waiting list.

However, docs says:

a delayed job will not be processed directly, instead it will be placed at the beginning of the waiting list and processed as soon as a worker is idle.

[manast]

When a delayed job is moved back to the wait list it is placed to the head of the list, i.e. it will be next to be processed... of course if other delayed jobs are moved to the wait list after the first one then they will move the first one back in the queue.

[kietheros]

@manast
I think this is a bug. You can use the above test case to reproduce this bug.

[manast]

What is the bug?

[kietheros]

Your expected results: [ j5, j4, j3, j2, j1, j0, d0, d1, d2, ...]
My test results: [d1, d0, j100, j99,... j4]

The bug is delays jobs were moved to the left of wait list instead of the right as in docs.
Test code to reproduce the issue in the first comment.

[manast]

Create an issue with minimal test code that reproduces it and I can look into it.

[manast]

The code above does not reproduce what you state btw... You should not use the rate limiter in order to reproduce the bug that you are stating.