Scaling and load testing

DOMjudge has been used in (very) large contests and can handle 1000+ teams and large sets of problems. Of course enough resources are needed.

The domserver can be scaled horizontally if wanted when the instances use the same database and there's a shared storage of PHP sessions (e.g. the same database, or redis or any of the other ways to create shared PHP sessions). See https://symfony.com/doc/current/session/database.html for general instructions on how to set up shared sessions in Symfony, the framework DOMjudge uses.

For DOMjudge load testing with gatling, see: https://github.com/ubergeek42/domjudge-gatling

Some tuning options:

• https://github.com/DOMjudge/domjudge/blob/master/etc/domjudge-fpm.conf.in#L15 - Adjust the max_children value in the fpm configuration. This value is based on the amount of memory your server has, divided by the memory usage of the domjudge php process. You can find the amount of memory being used per fpm worker process by looking in htop at the RES column(or rss column from ps). The comments in the file suggest it uses around ~25MiB, but ~70MiB seems to be more accurate these days. So if you have a 16GiB server, you should set this to something like: 15GiB/70MiB ~= 220. You need to make sure to leave some headroom for the rest of the processes on the server(and will need to allow even more room if you're running your database on the same host).

  • You need at least one worker per concurrent connection you expect to handle. Remember that requests from judgehosts count(they check in every ~10s), as well as all requests from team machines/spectators/jury/admins/etc. Most DOMjudge webpages automatically refresh every 30s(Scoreboard, team page, etc).
  • If you have ~1000 users, you're looking at a lower bound of around ~33 requests/second just based on the default 30s refresh of the scoreboard pages.
  • Your mysql server's max_connections value should probably be at least as large as the total number of fpm worker processes as well.

• In the server block of nginx, you may want to enable caching file descriptors/metadata. Something like the following has been used in the past:

  # Cache file descriptors and metadata, which greatly speeds up file access.
  # However, it means that the server won't react immediately to changes on disk,
  # but this is not an issue for DOMjudge as the filesystem is static
  open_file_cache          max=1000 inactive=20s;
  open_file_cache_valid    30s;
  open_file_cache_min_uses 2;
  open_file_cache_errors   on;
  

• Enable caching of static content(If using nginx). In the location block, you may want to turn on caching for static files. This will help reduce the number of hits to nginx; and allow offloading static content to a CDN if you're using one.

  # enable caching for static files
  location ~* \.(jpg|jpeg|png|css|js)\$ {
    access_log	off;
    log_not_found	off;
    expires		6h; # adjust as needed
  }