Total job count for building packages in parallel can get quite high.

[MSoegtropIMC]

Opam builds several independent packages in parallel if multiple packages are given in one opam install command. This is good, but the thread count can get excessively high because for each individual job "-j%{jobs}%" expands to -j15 on my machine. I think when opam runs several package builds in parallel, it should distribute the available jobs over these package build processes. So in my example on package build should get -j8 and the other one -j7.

My laptop has 32GB of RAM but swaps quite a bit for some multi package opam install commands.

Opam version: 2.0.7.

[dra27]

Indeed - the problem is that opam assumes non-parallel building of each package (which has never really been the case, given that ocamlbuild is also a parallel build system).

The solution is to implement make's jobserver, as @emillon mentioned in #4283 (comment) and also update Dune to support it.

[MSoegtropIMC]

Thanks for the pointers! I will see if it helps as a workaround to wrap the opam command in a make call. The full list of opam packages I need for my work environment takes more than one hour to build, so some optimization is worth the effort.

[dra27]

That won’t help - I was linking it to the other discussions ... it’s work which needs to be done!

[MSoegtropIMC]

Well the support in dune needs to be done and support for a job server in opam needs to be done but I don't see why it shouldn't be possible to feed the information of the job server from the outer make to an inner make through opam. Is the issue that the job server pipe is communicated via file handle numbers and opam does not hand these over to a child make? Otherwise it is just the environment variable the outer make creates.

[dra27]

I guess it might work if you use + on the opam invocation, although I confess I’ll be very surprised if it gets to the inner make! (It is, however, more worth a try than I initially thought!)

[MSoegtropIMC]

Yes, it is a bit unfortunate that make doesn't use named pipes - this would work regardless of file handle inheritance during child processes creation - and also work equally well on Windows and posix. I will give it a try and keep you posted.

I am btw. working on huge opam based Coq builds and since the Coq compiler easily takes more than 1 GB of memory per instance, one quickly gets into deep trouble when opam+make start 100 of these. On the other hand the compiles take long and have many serializing dependencies, so building several independent modules in parallel is a good idea. A job server would be an ideal solution for this.