Skip to content

Dev Worker Queues

Jump to bottom
RaghuSpaceRajan edited this page Mar 28, 2022 · 10 revisions

What are worker queues?

As a competition organizer each submission in your competition on Codalab has to be run on some machine. Codalab has many machines already provided on the default queue, but they may not meet certain needs for a competition you are running. Maybe you have a petabyte of data to process, for example, and it's not practical to transmit this for each competition.

You can add a custom queue here, connect your worker to it, and run submissions just for your competition or make your worker queue public so anyone can use it.

How do I setup a worker to process submissions?

  1. Install on your server docker and git
  2. Use BROKER_URL to the desired URL from the table below:

Replace <queue broker url> in the following command with your BROKER_URL

Note: this will make a /tmp/codalab directory

mkdir -p /tmp/codalab && docker run \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /tmp/codalab:/tmp/codalab \
    --env BROKER_URL=<queue broker url> \
    --env BROKER_USE_SSL=True \
    -d \
    --restart unless-stopped \
    codalab/competitions-v1-compute-worker:latest
Clone this wiki locally