Bootstraping ignores make jobserver

[jefferyto]

I'm currently working on enhancing the Rust package (currently 1.72.0) in OpenWrt (the build system is based on buildroot). After ensuring the x.py command is prefixed with + and that MAKEFLAGS is set correctly, bootstraping continues to use all of my CPUs instead of the -jN jobs from the make command line.

I have tried patching out these lines that remove MAKEFLAGS from the environment for cargo; the MAKEFLAGS value does reach cargo but cargo continues to set up its own jobserver instead.

I can see that removing MAKEFLAGS from the environment was deliberate (as noted in #50629 (comment)), but it would be great if building rustc can cooperate with the rest of the build system by using the same jobserver.

[onur-ozkan]

I can see that removing MAKEFLAGS from the environment was deliberate (as noted in #50629 (comment)), but it would be great if building rustc can cooperate with the rest of the build system by using the same jobserver.

Unfortunately, due to numerious problems achieving this 100% during the bootstrapping process is not possible. However, there is a workaround provided in this PR: #116108, which allows you to set the cargo jobs to the value provided to Make using -jN, similar to how it's done with x.py -jN.

bootstraping continues to use all of my CPUs instead of the -jN jobs from the make command line.

The PR linked above should resolve this problem.

[jefferyto]

However, there is a workaround provided in this PR: #116108, which allows you to set the cargo jobs to the value provided to Make using -jN, similar to how it's done with x.py -jN.

Thanks but that doesn't solve the issue. When I do a build in OpenWrt, I am often building many packages, not just Rust. If we just give the same -jN value to Rust, say -j4, then the overall build system will think there are 4 jobs available and Rust will also think there are 4 jobs available. Then potentially the overall build system will be building 3 other packages and Rust will be using 4 jobs, i.e. total of 7 jobs. This is why Rust should use the same jobserver as make.

[onur-ozkan]

I thought you were only building rust from make, I see now. Thanks for clarification.

I have tried patching out these lines that remove MAKEFLAGS from the environment for cargo; the MAKEFLAGS value does reach cargo but cargo continues to set up its own jobserver instead.

I assume the patch was removing those lines, right?

[jefferyto]

I assume the patch was removing those lines, right?

Yes. I removed the lines from bare_cargo() not rustdoc_cmd() though (the commit you referenced in #116108 (comment) was for rustdoc_cmd()); I assume rustdoc_cmd() is only invoked when building rustdoc.

[onur-ozkan]

Then this

the MAKEFLAGS value does reach cargo but cargo continues to set up its own jobserver instead.

means the real issue is more related with cargo itself.

[onur-ozkan]

We can remove

rust/src/bootstrap/builder.rs

Lines 1246 to 1248 in 70a7fe1

	// Remove make-related flags to ensure Cargo can correctly set things up
	cargo.env_remove("MAKEFLAGS");
	cargo.env_remove("MFLAGS");

lines once

the MAKEFLAGS value does reach cargo but cargo continues to set up its own jobserver instead.

is fixed from cargo side(I haven't tested it, so I added the needs-triage label).

[jefferyto]

means the real issue is more related with cargo itself.

I'm able to have cargo use the same make jobserver (to build other Rust packages) after rustc is compiled/installed (cargo is called directly from the Makefile), so I would say that bootstrap is at least involved in this issue, if not the main cause.