rustbuild: copy libs for stage1 from stage0 when using incremental and enable-local-rust

[nikomatsakis]

When building stage 2, we could copy the libstd etc that we built from stage1 instead of rebuilding (it has the correct metadata format). If we are using --enable-local-rust, we could do this even with --stage 1 for the output from stage0 (presuming that you haven't locally changed the metadata). The downside is that you lose the testing of running the compiler you just built on libstd.

This is orthogonal from incremental, but related in that it is a good way to improve local turnaround time, particularly since we can't use incremental after stage0 (since the compiler itself changed underfoot). As such, I'm forking this issue off from #37929.

cc @alexcrichton, who suggested this

[michaelwoerister]

Related to this issue is also that the old build system split building libraries while rustbuild doesn't:

• For running most test suites (run-pass, run-fail, compile-fail, incremental, ...) the old system only built libraries from libcore to libstd (roughly), which takes about 40 seconds on my machine.
• Only for running additional test suites (run-pass-fulldeps, run-make, ...) it built the rest of the libraries (syntax to librustc_driver, the heavy weights). This takes a lot longer, at least 12 minutes.

This makes a typical cycle of my current workflow, consisting of changing some part of the compiler and verifying the change via the tests, go from 5 minutes to 16 minutes. And rustbuild --incremental can only help with the first 4 minutes in both cases. So this optimization is rather important. Incremental + copying libraries could bring the above case down to about 30 seconds.

[aidanhs]

A brief outline of the stage system as I understand it:

• stage0-in - downloaded from the internet (or the local rust)
• stage0-out - std, test, rustc (created with stage0-in, latter two use the stage0-out std) (all this can be incremental, as stage0-in doesn't change)
• stage1-in - copied stage0-out std and rustc
• stage1-out - std, test, rustc (created with stage1-in)
• stage2-in - copied stage0-out std and rustc
• stage2-out - std, test, rustc (created with stage2-in)

I'm not sure that it's quite right to consider incremental builds as orthogonal to the stages system (i.e. the current implementation of it just being a flag). For example, the whole point (if I understand correctly) of having stage0-out/stage1-in creating stage1-out/stage2-in, then using this to create stage2-out, is so that you can verify that stage1-out and stage2-out are the same. If you don't care about this verification (e.g. you're in incremental mode and copying std of stage0-out all the way through) there's no point in going beyond creating stage1-in - you're just adding extra time. As a result, I would say stages built in incremental mode when doing library copying are better presented to the user as distinct things (e.g. stageN-incremental) though internally they could be represented the same. This makes it clearer that you are not 'just enabling incremental', allows a descriptive warning message if someone tries stage2-incremental and heads off questions why incremental isn't the default.

Slightly orthogonal is the observation that it would actually be nice to be able to have a target that stops at stage1-in, as that's the first point where you have a compiler you can rustup toolchain link for testing a change and it fully benefits from incremental compilation. Unfortunately, doing x.py build --stage 0 compiles but doesn't assemble the pieces into a single directory, whereas x.py build --stage 1 goes through the stage1-out compilation. Perhaps there is a target and I missed it?

In addition, I believe @michaelwoerister's case may be helped by noting that it should actually be possible to run all the libstd tests immediately after creating std of stage0-out - there's no reason to build librustc at all if you've just made a change to (for example) libcollections, ./x.py test src/libcollections --stage 0 --incremental should just work. It doesn't at the moment though, see #38589

Please let me know if any of the above seems wrong, I've only recently been looking into the stages system so may have missed something.

[alexcrichton]

Ok I think there's a lot going on in this issue, so I want to try to help tease it apart. As currently titled I think that this issue is conflating two orthogonal issues.

First, one optimization we can do as part of the build is to just omit the third time we compile the compiler. We only need to fundamentally compile the compiler twice. That's covered by the second bullet in #38531 (sorta).

Second, with --enable-local-rust (or some other random --enable-* flag, I forget which) we're indicating that the local Rust compiler is 'new enough' that we can skip the first compile, meaning we only need to compile the compiler once.

In the latter case we can build the compiler once incrementally and be done with it. For the former case we can continue to build the first compiler incrementally but must continue sequentially and not incrementally from that point.

So in that sense I think this issue may be subsumed (or considered a sub-bug) of #38531.

---

@michaelwoerister what you're mentioning sounds quite worrisome! When running suites like run-pass rustbuild should not require compiling the compiler in a stage, you should be able to start executing that promptly after libtest finishes building.

Could you describe what you're doing which is taking too long?

---

@aidanhs I think what you've said is mostly right. Note, though, that --incremental doesn't imply copying stage0-out everywhere. It just means that generation of stage0-out is itself incremental (e.g. passing -Z incremental=...).

[nikomatsakis]

@aidanhs I'm not sure if this is the same as what @alexcrichton said, but I was thinking that it's orthogonal in that you can copy the libs from stage N-1 to stage N, where N is the "final stage". This would mean copying output from stage 1 to stage 2 if you're building with beta as the snapshot, but if your snapshot is "new enough", you could copy stage 0 to stage 1 (and consider stage 1 as the final stage). This isn't really about incremental compilation so much as having a "new enough" snapshot, which we indicate with --enable-local-rebuild --enable-local-rust. A new enough snapshot is, I think, one that generates compatible metadata. (Make sense?)

[aidanhs]

@alexcrichton

I think it's important to talk about scenarios. If you're locally developing, I can't think of any reason to build anything more than once (i.e. no further than stage1-in). But for (say) merging a commit it's important to make sure the compiler is self hosting so you do it twice (i.e. no further than stage2-in). If you want extra verification then you can do it a third time and compare the outputs (perhaps for creating releases?somewhat niche anyway, so that second bullet seems sane).

With that in mind, what scenario is being addressed by skipping a compile/when is this compile step skipped? Local development is only compiling once anyway (which is incremental), and checking a commit for merge must compile a compiler with itself (the second compiler) to verify self-hosting (which cannot be incremental because it's changing).

Note, though, that --incremental doesn't imply copying stage0-out everywhere. It just means that generation of stage0-out is itself incremental (e.g. passing -Z incremental=...).

Yes, sorry, I was vaguely referring to the intention of this issue to copy libraries around, rather than a description of --incremental as it stands right now.

---

@nikomatsakis I've realised that this issue and your comment above makes sense if I agree that a compiler build should/could be skipped somewhere. As it stands, I'm not sure I do! There are up to three possible builds of the rust compiler in the stages system, each with a distinct purpose:

• stage0-in (local, beta, etc) -> stage0-out - creates a functioning compiler from the checked out source
• stage1-in -> stage1-out - verifies that the compiler represented by the checked out source is self-hosting
• stage2-in -> stage2-out - verifies that the compiler generates itself identically

If you just want a functioning compiler (e.g. for local dev) then you stop before compiling stage1-out, no copying libraries necessary (nor local snapshots, beyond them having better incremental support). In fact, copying libraries or skipping compiles defeats the intent of the steps as stated above!

I may well have misunderstood the purpose of one of the stages, in which case I am keen to understand and document it somewhere for future explorers.

[nikomatsakis]

If you just want a functioning compiler (e.g. for local dev) then you stop before compiling stage1-out, no copying libraries necessary (nor local snapshots, beyond them having better incremental support).

Hmm, this might be a bit imprecise. One wrinkle is that, to use compiler X, you need to have a version of libstd etc that has metadata compatible with compiler X. Strictly speaking, the only official guarantee of compatible metadata is that libstd etc were produced by compiler X. But sometimes, in practice, any compiler Y that is "close enough" to X will do. Hence the desire to copy libraries.

But I think in the grander picture you are right, and I would sort of like a declarative way to pick "a role" for rustbuild. --incremental is sort of in the right direction (as it suggests that your "role" is to get a functioning compiler from the checked out source, and that's it), but perhaps we can do better. I seem to recall @brson advancing a similar notion in the past. I think we identified other roles too: e.g., testing libstd.

[alexcrichton]

@aidanhs Ah I think @nikomatsakis may have answered as well, but I'll try to help clarify as well. All compilers are basically useless unless they have a libstd to link with as well. So once you actually build a compiler, you still need to get a standard library from somewhere to link against.

Now the compiler changes metadata format, ABI, etc, from time to time. This means that we can't just pick any old standard library off the shelf. Consequently, the only standard library guaranteed to work (at least as understood historically) is one produced by the compiler itself. Recently we've had the revelation, however, that while true we can actually make a looser claim that a compiler is guaranteed to be compatible with any library produced by a compiler running the same source code.

That may seem a little odd, but if we go through the three compilers again:

• stage0-in - the source code for this compiler is some old historical beta
• stage1-in - the source code for this compiler is what's currently in the source tree
• stage2-in - the source code for this compiler is the same as stage1-in

So that means that the libstd produced by the stage1-in compiler is actually compatible to be used by the stage2-in compiler. That's precisely what #38631 does as well! No more artifacts are produced during stage2-out, they're just copied from stage1-out.

---

To dive into some of your specific points, though:

I can't think of any reason to build anything more than once (i.e. no further than stage1-in).

Ah so this is the problem about standard libraries. Once you've created the stage1-in compiler, that compiler has no libraries to work with. It can't use the libraries of stage0-out because they were produced by a compiler with different source code. Similarly it can't use stage0-in libraries because they were also produced by a compiler with different source code.

So to be minimally useful the stage1-in compiler would at least have to compile the standard library (e.g. some of the artifacts in stage1-out). To end up running the full test suite, though, you'll have to go all the way for plugins and produce everything.

In general, you're right though. Most dev shouldn't need to go past stage1-in + stage1-out (libstd). That's actually normally what I work with when doing compiler development!

But for (say) merging a commit it's important to make sure the compiler is self hosting so you do it twice (i.e. no further than stage2-in).

Note that this actually isn't sufficient for self hosting. To prove we can self-host we actually need to go all the way through to stage2-out. Completing stage1-out means that a historically compiled compiler can recompile all source code. But the guarantee that we want to preserve is that the current source code can compile itself. The stage2-in compiler is the first compiler produced by the current source code, so to fully prove the bootstrap chain we have to then use that compiler to compile all source code again.

Put another way, each stage gives us proofs like:

• stage0-out - proves that the code compiles under an older compiler (e.g. last stable release)
• stage1-out - proves that when compiled by an older compiler, the current source code can produce another compiler
• stage2-out - proves that the current source code can compile the current source code

Consider, for example, a codegen bug. We accidentally defined a + b as a - b. That wouldn't actually show up until we run the stage2-in compiler. The stage1-in compiler wouldn't suffer from this bug, but all programs it produces will suffer this bug.

Does that make sense? This is why in #38631 although we'd turning off a whole stage by default we have a builder that still does the full bootstrap.

With that in mind, what scenario is being addressed by skipping a compile/when is this compile step skipped?

Ah so #38631 may provide more context here. Specifically, though, the stage2-out artifacts are not produced but rather copied from the stage1-out location. This means that all compiler development can avoid an extraneous stage of building the compiler.

Local development is only compiling once anyway (which is incremental), and checking a commit for merge must compile a compiler with itself (the second compiler) to verify self-hosting (which cannot be incremental because it's changing).

I think I've answered this above as well, but please feel free to ask more questions if I haven't!

[michaelwoerister]

@alexcrichton

Could you describe what you're doing which is taking too long?

That particular problem seems to be gone in the current version of rustbuild. Maybe that was fixed by #38631? Or maybe I was just assuming that x.py build --stage 1 would do the same as make rustc-stage1 and didn't think of actually verifying that assumption. Thanks, anyway!

[aidanhs]

@nikomatsakis @alexcrichton many thanks to you both for your patience and help here and on IRC. I now believe I understand the stages system and the rust compiler build system.

The piece I was missing was that there are two stdlibs involved in running a compiler - 1) the stdlib .so files the compiler itself has been linked to, where metadata isn't generally used (maybe plugins?) and 2) the stdlib it links compiled programs to (the sysroot) where metadata must be compatible.

[steveklabnik]

Triage: is any of this relevant anymore?

[jyn514]

@steveklabnik it's relevant in the senses that building stage 1 libstd can still be a roadblock: https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html?highlight=hacky#building-the-compiler. Copying stage1 std to stage2 std is now done automatically, so that bit works:

$ x.py build --stage 2
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)

The thing left is that @nikomatsakis suggested using stage0 std for stage1, which seems difficult to me - you'd need an extremely recent version of nightly as the ABI for std is not stable and can break at any time. Since bootstrap uses beta by default, which can be up to 6 weeks out of date, I don't think it currently makes sense to enable uplifting stage0 std.

I'm not sure how incremental is related to this.

[jyn514]

cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Building.20rustc.20with.20beta.20libstd / rust-lang/compiler-team#619

[jyn514]

I'm going to close this - I don't think the ~30 seconds or so we'd save by not recompiling stage0-std are worth forcing people using --enable-local-rust to think about whether std is compatible between stage 0 and stage 1, and all other parts of this issue have since been fixed.

See also rust-lang/compiler-team#619, which would make this problem go away entirely by not building stage 0 std before building the compiler.