build: Add the parallel feature, set -j arg based on NUM_JOBS

[ParkMyCar]

This PR speeds up builds of protobuf-src for me from ~5 minutes to ~1 minute.

There's probably a higher level issue that I haven't dug into here yet, the actually cmake command that gets run, without this PR, is:

MAKEFLAGS="-j --jobserver-fds=8,9 --jobserver-auth=8,9" cmake --build . --target install --config Debug

This should parallelize the build with Cargo's jobserver (I think?), but something seems to go wrong and at least locally, after ~30 seconds I only see one instance of clang running.

With this PR the cmake command is updated to:

MAKEFLAGS="-j --jobserver-fds=8,9 --jobserver-auth=8,9" cmake --build . --target install --config Debug -- -j12

Further, looking at the logs from cmake this seems to disable jobserver mode:

[protobuf-src 2.1.0+27.1] make: warning: -jN forced in submake: disabling jobserver mode.

This isn't ideal, but something seems to be going wrong with Cargo's job server which is seriously slowing down builds.

[def-]

My main concern with just parallelizing a lot is that this can cause OoMs.

I think cmake might just ignore the MAKEFLAGS. How about adding the entire -- -j --jobserver-fds=8,9 --jobserver-auth=8,9 at the end instead?

[benesch]

Hrm, seems like some upstream issues about this:

• Parallel builds break when combined with cc-rs using the "parallel" feature rust-lang/cmake-rs#172
• New jobserver type in GNU Make 4.4 rust-lang/jobserver-rs#47

Not sure what the right fix is, but concur with Dennis that disabling the jobserver and allowing parallelism has led to OOMs in CI during builds in the past.

[petrosagg]

Is there a way to have this for builds on dev machines? It's rare that this crate needs to be rebuilt but not THAT rare, especially if I'm jumping around branches etc

[benesch]

Is there a way to have this for builds on dev machines? It's rare that this crate needs to be rebuilt but not THAT rare, especially if I'm jumping around branches etc

It's still dicey. We've seen compiling Materialize OOM on dev machines in the past when we haven't been careful to let one single jobserver be responsible for managing all the parallelism.

The jobserver protocol isn't all that complicated. I'm hopeful that it might be possible to track down whatever the root cause here is without too much trouble.

[ParkMyCar]

Concerns about memory usage totally makes sense! I'm going to close this out for now since this PR isn't the right approach, but also I don't have the cycles at the moment to investigate the upstream jobserver issue. As a workaround folks building Materialize can use Bazel which handles this more gracefully.