Skip to content

Litejob guide

Jump to bottom
Mohammad A. Ali edited this page Mar 26, 2023 · 2 revisions

The Litejob guide

Litejob, a part of Litestack, is a high performance, powerful yet very simple background job processing engine for Ruby and Rails applications.

Litejob, as other components in Litestack is built on top of SQLite. It uses the embedded database engine to enqueue and dequeue jobs for processing. As a result, Litejob is a very low maintenance job processing system. There is no need to setup/maintain/monitor any service aside from the application that integrates Litejob.

Careful consideration went into the design of the database schema and SQL queries to deliver the best performance possible for Litejob. A lot of effort also went into ensuring that the background job de-queuing thread/fiber is as cooperative as possible with other running threads/fibers.

Features & Roadmap

  • Named queues
  • Delayed execution
  • Retry failed jobs
  • Garbage collect dead jobs
  • Rails ActiveJob integration
  • Thread safety
  • Async/Fiber Scheduler integration
  • Graceful shutdown
  • Fork resilience
  • Polyphony integration
  • Timeout support for job execution

How to use Litejob?

Native Interface

For using the native Lite job interface all you need to do is define your job class, include the Litejob module and implement perform, as in the following example:

Defining jobs

# note that we only need to require litestack
# you could still do require 'litestack/litejob'
require 'litestack'
class AddJob
  include Litejob
  # select which specific named queue your jobs should go to
  self.queue = 'my_job_queue'
  # perform must be implemented and can have any number of parameters
  def perform(a, b)
    a + b
  end
end

Scheduling jobs

# to invoke your jobs asynchronously
AddJob.perform_async(1, 1)
# to invoke at a certain time
AddJod.perfrom_on(time, 1, 1)
# to invoke after a certain delay
Addjob.perform_in(delay, 1, 1)

Configuring Litejob

Litejob looks for a litejob.yml file in its working directory, the syntax and defaults for the file are as follows:

path: '/queue.db' # where the database file resides
queues:
  - [default, 1] # default queue with the lowest priority 
  - [urgent, 10, spawn] # this is not a default, a higher priority queue which will run every job in its own thread or fiber 
workers: 5 # how many threads/fibers to spawn for queue processing
retries: 5 # how many times to retry a failed job before giving up
retry_delay: 60 # seconds
retry_delay_multiplier: 10 # 60 -> 600 -> 6000 and so on 
dead_job_retention: 864000 # 10 days to keep completely faild jobs in the _dead queue
gc_sleep_interval: 7200 # 2 hours of sleep between checking for dead jobs that are ready to be buried forever
logger: STDOUT # possible values are STDOUT, STDERR, NULL or a file location

The db path should preferably be outside of your application folder, in order to prevent accidental overrides during deployment

The spawn option should not be used in a threaded environment (e.g. with the Puma web server)

Usage with Rails/ActiveJob

You can simply use the native interface in your Rails application, but if you want to specifically use the ActiveJob interface you can configure it as such in your environment file (e.g. production.rb)

config.active_job.queue_adapter = :litejob

Then you can use ActiveJob normally.

In the Rails environment, the litejob.yml configuration file should be located in the /config directory under the root Rails directory

Performance considerations

Lock contention

Litejob relies on a very low overhead queuing/dequeuing engine. That engine shows the best performance when fewer processes are competing for pushing to or popping from the queue. As a rule of thumb, if your processes are doing nothing but pushing and processing jobs as fast as possible, 8 parallel processes sound like a reasonable limit, after which lock contention will have a sizable overhead. In a web application of course, the processes will be doing a lot more activities as well, and hence the upper bound for optimal performance can be much higher.

Threads vs Fibers

Thanks to Litesupport, Litejob works seamlessly with both threads and fibers based on the environment it runs in. For performance though, fibers generally offer lower overhead. And specially with the "spawn" option that runs each job in a separate fiber, huge performance gains can be achieved for IO bound jobs.

Job size

Generally speaking, it is not recommended to pass really large parameters to jobs. Litejob uses a special type of SQLite3 tables that stores data within the nodes and the leaves of the B-tree (as opposed to just on the leaves for typical tables). It is best if a single database page, typically 4096 bytes, can store at least 4 jobs, meaning a job should be slightly less than 1KB for optimum performance. Larger job sizes will work without issues, but the performance profile will change.

Clone this wiki locally