RUSTC_WRAPPER is ignored for rustc -vV call

[RalfJung]

Problem

We are setting RUSTC_WRAPPER in order to intercept every invocation of rustc from cargo. However, it looks like cargo will call rustc -vV in any case, completely bypassing the RUSTC_WRAPPER env var. Only the RUSTC env var seems to be honored for the -vV call. This is causing us a bunch of pain right now since we need to figure out how to best intercept this -vV call.

The docs describe both RUSTC and RUSTC_WRAPPER in very similar terms, so I think this is clearly a bug -- both env vars should be effective for the same invocations.

We prefer to use RUSTC_WRAPPER over RUSTC because we need to intercept all things that cargo calls (including rustdoc, and running the actual binaries), and RUSTC_WRAPPER makes it easier to figure out what cargo called us for -- if we are invoked with the first argument being rustc, we know it was rustc that cargo wanted. (This is in the cargo-miri wrapper.)

[RalfJung]

The current behavior is annoying for both Miri and bootstrap, and probably other tools -- everyone has to overwrite both RUSTC_WRAPPER and RUSTC, which is quite surprising. At least in the case of Miri it is also fragile: what I'd like to do is set RUSTC to /cargo-miri-phase-rustc/if-you-see-this-some-build-script-forgot-to-apply-the-rustc-wrapper or something like that, so that when we get invoked as the wrapper we can easily tell that this is what happens. But of course that doesn't work because cargo then calls the non-existing path. This means we have to use a less reliable technique to detect when we are the wrapper. It also means we can't outright error when something invokes RUSTC without the wrapper, even though that is almost always a bug -- it is absolutely crucial that the wrapper is applied so that it can adjust the command-line arguments accordingly.

@ehuss this was previously attempted in #10887, but unfortunately rejected because more testing was requested. So it seems like passing the cargo test suite is not sufficient. I have never used sccache, so I don't even know what "using sccache" looks like, let alone testing it sufficiently. Is there documentation of what you think should be done? Is it sufficient to pass rustc bootstrap, or does that not use sccache? sccache is mentioned in src/ci, for whatever that is worth.

It seems hard for me to imagine that a wrapper would work with $WRAPPER rustc --print sysroot (which is wrapped!) but not with $WRAPPER rustc -vV. So I don't quite understand why you think this is so risky.

[ehuss]

To test with sccache, I would recommend going to https://github.com/mozilla/sccache and reading the instructions. You just need to install the sccache binary (cargo install sccache usually works), and that page also explains how to set RUSTC_WRAPPER to wherever you installed it. Then, just go try building a few projects with a cargo including the change from #10887 and make sure it still works, and the caching works.

rustc does not directly use sccache for building Rust code, it is only used for caching the llvm build. I think some people use it for local development, and I believe all you have to do is set RUSTC_WRAPPER.

The concern is that sccache parses the rustc command-line, and there have been several instances in the past where small changes of the options have broken sccache. It probably works, but it is worth checking.

[RalfJung]

FWIW bootstrap also has trouble with this.

[epage]

Copying over from #14385 (comment)

As described here (but admittedly without going into much detail), the end users I am concerned about are rustc bootstrap and cargo-miri. They both set RUSTC_WRAPPER. rustc bootstrap uses the wrapper to decide which stage's compiler to invoke. Things would obviously go quite wrong if cargo got the wrong rustc version here. To achieve that, bootstrap currently sets both the RUSTC and RUSTC_WRAPPER env vars, but that then requires awkward hacks in the rustc binary that it uses for both of those env vars because the binary has to auto-detect whether it got invoked as $RUSTC or as $RUSTC_WRAPPER $RUSTC. That hack also got linked in the issue I referenced above. cargo-miri has similar hacks. (Sadly Miri will likely have to keep these hacks forever as there are too many build scripts out there that invoke rustc without accounting for the wrapper. But that is no excuse for cargo itself to do this incorrectly, and rustc bootstrap may get away with dropping the hack since it more tightly controls which crates it is run on.)