Skip to content

Dev Worker Queues

Jump to bottom
Isabelle Guyon edited this page Oct 4, 2017 · 10 revisions

What are worker queues?

As a competition organizer each submission in your competition on Codalab has to be run on some machine. Codalab has many machines already provided on the default queue, but they may not meet certain needs for a competition you are running. Maybe you have a petabyte of data to process, for example, and it's not practical to transmit this for each competition.

You can add a custom queue here, connect your worker to it, and run submissions just for your competition or make your worker queue public so anyone can use it.

How do I setup a worker to process submissions?

  1. Install on your server docker and git

  2. git clone [email protected]:codalab/codalab-competitions.git && cd codalab-competitions

  3. Edit .env to set BROKER_URL to the desired URL from the table below: Now your .env file should have this line

    BROKER_URL=pyamqp://6ad9ac58-88e3-4a22-9be7-6ed5126ef388:40546f9d-0f2e-4413-a6b2-a1d86cad2b30@localhost/37324ab2-ee78-4e8d-a6ee-6089a159d253. KNOWN BUG: Sometimes the server IP address or URL is replaced by localhost, like in the example above. If you get this, substitute localhost by the IP address or URL of your server.

  4. docker-compose up compute_worker -d

  5. Make a submission to your competition and check logs with docker logs -f worker_compute to confirm it is working

Clone this wiki locally