Home

forerunner

forerunner is a flexible distributed job queue framework that is flexible enough to fit any need. This wiki serves as a reference for plugins and best practices for use.

Overview

Forerunner is made up of 4 main parts:

1. Store - permanent storage for jobs state and results
2. Queue - ephemeral storage for jobs that need to be processed
3. Manager - central logic for distributing and handling job state
4. Worker - external processes that perform the actual work

Each of these components are extendable and can be dropped into just about any system. In general, the queue will be either a in-memory system or a redis plugin. The store should be PostgreSQL or something comparable. The manager can be extended through the use of plugins to enable more complicated functionality around how jobs enter and leave the framework.

Definitions

• Task - a well defined, individual operation that takes place on a worker (ie fetch an HTML page).
• Job - a series of tasks to be executed in order on one worker. Consists of at least one task.
• Worker - a server is able to execute a set of defined jobs. The more the better!