actinia config options

Redis Queue for Jobs

There are different options for distributing jobs within an actinia setup. They can be controlled with the config option QUEUE.queue_type. The options are:

local

[QUEUE]
queue_type = local

The actinia instance which receives the job will process it. In this setup, multiple actinia instances can run next to each other and the load balancer from the installation environment (docker-compose, kubernetes, ...) is responsible to distribute the jobs. When a job is send asynronously to /processing_async or /processing_async_export endpoint, actinia will start the job itself.

per_job

[QUEUE]
redis_queue_server_url = 127.0.0.1
redis_queue_server_password = pass
worker_prefix = job_queue
queue_type = per_job

In this case, actinia will receive the job and write it into the redis queue. An actinia worker is needed to process the job. The worker needs to listen to the queue. The name can be found in the actinia response at "queue":

{
	"accept_datetime": "2023-09-02 16:47:12.733873",
	"accept_timestamp": 1693673232.733873,
         ...
	"queue": "job_queue_resource_id-0a151f6c-91c5-478c-84b8-57fb4ce5f052",
	"resource_id": "resource_id-0a151f6c-91c5-478c-84b8-57fb4ce5f052",
	"status": "accepted",
	"user_id": "demouser"
}

As for each job a new queue is created, a new worker needs to be started explicitly. It is under current development that actinia takes care of this itself, but up to now a manual step is needed:

actinia-worker -c /etc/default/actinia --quit job_queue_resource_id-0a151f6c-91c5-478c-84b8-57fb4ce5f052 

per_user

[QUEUE]
redis_queue_server_url = 127.0.0.1
redis_queue_server_password = pass
worker_prefix = job_queue
queue_type = per_user

See above, same as per_job. DIfference is, that not for each job a new queue is created but for each user.

actinia-worker job_queue_lina -c /etc/default/actinia --quit

redis

[QUEUE]
redis_queue_server_url = 127.0.0.1
redis_queue_server_password = pass
worker_prefix = job_queue
number_of_workers = 1
queue_type = redis

In this case, the job is enqueued as well. Difference is that the names of the queue are predictable and build from worker_prefix and count. In this case, the workers can run and wait for jobs

actinia-worker job_queue_0 -c /etc/default/actinia

More details from actinia-worker:

#/src/actinia_core # actinia-worker --help
usage: actinia-worker [-h] [-c CONFIG] [-q] queue

Start a single Actinia Core custom worker listening to a specific queue.It
uses the logfile settings that are specified in the default Actinia Core
configuration fileor a file specified by an optional path.

positional arguments:
  queue                 The name of the queue that should be listen to by the
                        worker

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        The path to the Actinia Core configuration file
  -q, --quit            Wether or not the worker should exit when the queue is
                        emptied.