Queue

Queue

In forerunner the queue is a module that allows forerunner to iterate quickly over jobs that still need to be processed. The queue can be as simple as an array in javascript, but is abstracted into its own module to allow for systems such as redis to be used. While it is called a queue module, FIFO order is not required for forerunner to function properly.

Writing your own queue

First of all, make sure that there is not another forerunner queue that meets your needs :)

The only requirement placed on your custom queue is that it passes the queue tests which define how forerunner is expecting the queue to operate. If you pass the tests, then your queue will work just fine!

Methods to implement

push(jobId, jobType, jobData, callback(err))

This will push a new job into the queue. The id, type, and data inputs will need to be available when the job is pulled later on. The callback takes a single error argument. If this error is not falsy then forerunner will assume the job was not placed into the queue.

each(count, eachFn(id, type, data), callback(err))

This will remove a count number of elements from the queue and call the eachFn for each of those elements. If there are less than count objects in the queue then only call eachFn for the number of objects in the queue. When the all elements have been iterated over, callback. It is suggested that you use async's each method to handle the two functions.

requeue(jobId, jobType, jobData, callback(err))

This operates very similar to push. The only difference is that a requeued job should have higher priority than a pushed job since forerunner pulled a job from the queue, but was unable to pass it off to a worker. If FIFO is trying to be maintained, this would be an Array.unshift operation.

empty(callback(err))

Empties the queue, and calls back with an error if that failed. Remember that the queue object is assumed to be ephemeral (even though it might not be) so emptying the queue will not result in the loss of jobs. This is indented be used in conjunction with the options.loadFromStore flag.