parallelize locale compilation

[KevinMind]

relates to: mozilla/addons#1992

Description

Run our locale compilation in parallel (with xargs) to improve the docker build speed.

Context

Serial execution:

real 1m11.109s
user 1m3.415s
sys 0m4.928s

Parallel execution: (not using)

real 0m18.702s
user 1m36.202s
sys 0m14.430s

Xargs execution: (winner)

real 0m23.595s
user 1m16.155s
sys 0m8.179s

We will save an average of 50-60 seconds per build!

Testing

You can shell into the docker container

make shell

And then run the compile command

time DJANGO_SETTINGS_MODULE='olympia.lib.settings_base' locale/compile-mo.sh locale

[diox]

I like the PR, but parallel is a bit heavy (at least the debian package is)...

# apt install parallel
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  cron exim4-base exim4-config exim4-daemon-light guile-2.2-libs libevent-2.1-6 libgc1c2 libgnutls-dane0 libgsasl7 libkyotocabinet16v5 libltdl7 liblzo2-2 libmailutils5 libmariadb3 libntlm0
  libprocps7 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libsensors-config libsensors5 libunbound8 libwrap0 mailutils mailutils-common mariadb-common procps psmisc sysstat
Suggested packages:
  anacron logrotate checksecurity exim4-doc-html | exim4-doc-info eximon4 spf-tools-perl swaks dns-root-data lm-sensors mailutils-mh mailutils-doc isag
The following NEW packages will be installed:
  cron exim4-base exim4-config exim4-daemon-light guile-2.2-libs libevent-2.1-6 libgc1c2 libgnutls-dane0 libgsasl7 libkyotocabinet16v5 libltdl7 liblzo2-2 libmailutils5 libmariadb3 libntlm0
  libprocps7 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libsensors-config libsensors5 libunbound8 libwrap0 mailutils mailutils-common mariadb-common parallel procps psmisc
  sysstat
0 upgraded, 30 newly installed, 0 to remove and 0 not upgraded.

Could we find something lighter that doesn't force us to pull all these packages ?

[diox]

Simplest option might be to use xargs with a fixed number of processes (-P) that makes sense for our CI environment (4 ?)

[diox]

https://www.gnu.org/software/parallel/parallel_alternatives.pdf lists some alternatives - we don't need all the extensive feature set it supports

[KevinMind]

Using xargs gives us

real 0m23.595s
user 1m16.155s
sys 0m8.179s

Not quite as good, but still a significant improvement.

Just a note, A PR I plan to create in this series will add a 2 stage build to our docker image. In the first stage we will do all the compilation and file manipulation stuff, then we can copy over the compiled assets to the second stage.

Theoretically that would enable us to use parallel without including it in the image. You'd still need to download it "once" (after we get the good good caching) but it wouldn't actually land in the image.

I'm happy to use xargs instead, but just thought it was worth mentioning we can split our docker dependencies into dev/prod just like we do with pip and npm.