Tracking Issue for removing python as the main bootstrap entry point #94829
Comments
jyn514
added
the
C-tracking-issue
rustbot
added
the
T-bootstrap
Helps with rust-lang#94829. This doesn't solve "bootstrap uses beta rustc but nightly rustfmt", but in practice there should be very little difference between two. Users can still override the toolchain used with `cargo +nightly` or similar.
Distros generally need offline builds, so I'm not sure what role you see for bootstrap.py there. For most builds I'm just pointing to the |
Ah, interesting. I'm a little confused why we ever had a python entry point then - @Mark-Simulacrum do you remember the original goal? Was it intended for individual users trying to build from source? I'm ok with regressing that a little bit as long as people can still install rustup from source and then build bootstrap ... |
|
bootstrap.py was introduced when rewriting the build system from make to rust in 046e687. Back then rustup (it was still called multirust at that point I believe) wasn't really common I believe. |
|
The python wrapper supports us in a number of ways (or did): submodule initialization (needed before first Cargo invocation due to workspace resolution), downloading and checking hashes on a specific beta/stable (sort of replaceable with rustup, but historically our needs are a little more complex such as with rustfmt being a separate toolchain), figuring out what the platform target triple is... It also does more "mundane" things like permitting a single interface to rustbuild without cargo run -- ... prefixing every command, which to me feels at least not obviously better than x.py. (And configuring that initial Cargo invocation with our linting requirements etc) |
|
Cargo allows adding aliases in |
We could have that unified interface with The other things are mostly done by rustup, or we could make them so - I guess the concern is for tooling in CI, which doesn't have rustup installed? We could make submodules fully managed by rustbuild if we took it out of the workspace I think. That doesn't seem like a terribly bad regression; it mostly just means that it won't show up in |
|
I have divided this up into some smaller tasks that can be done in parallel; see https://rust-lang.zulipchat.com/#narrow/stream/122652-new-members/topic/Help.20with.20RRIR.20bootstrap.20entrypoint for more info. @rustbot label: +E-mentor +E-help-wanted |
rustbot
added
E-help-wanted
…imulacrum Fix `cargo run` on Windows Fixes the following error: ``` error: failed to run custom build command for `bootstrap v0.0.0 (C:\Users\Walther\git\rust\src\bootstrap)` Caused by: process didn't exit successfully: `C:\Users\Walther\git\rust\target\debug\build\bootstrap-7757a4777dec0f86\build-script-build` (exit code: 101) --- stdout cargo:rerun-if-changed=build.rs cargo:rerun-if-env-changed=RUSTC cargo:rustc-env=BUILD_TRIPLE=x86_64-pc-windows-msvc cargo:rerun-if-env-changed=PATH --- stderr thread 'main' panicked at 'assertion failed: rustc.is_absolute()', src\bootstrap\build.rs:22:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace warning: build failed, waiting for other jobs to finish... error: build failed ``` The problem was that the `dir.join` check only works with `rustc.exe`, not `rustc`. Thanks `@Walther` for the help testing the fix! Helps with rust-lang#94829.
…ulacrum Fix `cargo run` on Windows Fixes the following error: ``` error: failed to run custom build command for `bootstrap v0.0.0 (C:\Users\Walther\git\rust\src\bootstrap)` Caused by: process didn't exit successfully: `C:\Users\Walther\git\rust\target\debug\build\bootstrap-7757a4777dec0f86\build-script-build` (exit code: 101) --- stdout cargo:rerun-if-changed=build.rs cargo:rerun-if-env-changed=RUSTC cargo:rustc-env=BUILD_TRIPLE=x86_64-pc-windows-msvc cargo:rerun-if-env-changed=PATH --- stderr thread 'main' panicked at 'assertion failed: rustc.is_absolute()', src\bootstrap\build.rs:22:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace warning: build failed, waiting for other jobs to finish... error: build failed ``` The problem was that the `dir.join` check only works with `rustc.exe`, not `rustc`. Thanks `@Walther` for the help testing the fix! Helps with rust-lang#94829.
Make it possible to run `cargo test` for bootstrap Note that this only runs bootstrap's self-tests, not compiler or library tests. Helps with rust-lang#94829.
Move `download-ci-llvm` out of bootstrap.py This is ready for review. It has been tested on Windows, Linux, and NixOS. The second commit ports the changes from rust-lang#95234 to Rust; I can remove it if desired. Helps with rust-lang#94829. As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that. `@rustbot` label +A-rustbuild
Distribute bootstrap in CI This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here: 1. Make it faster to build rust from source, both the first time and incrementally 2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information. There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage. Helps with rust-lang#94829.
…Simulacrum Make all download functions need only Config, not Builder This also adds a new `mod download` instead of scattering the download code across `config.rs` and `native.rs`. This is the simplest and also most bit-rotty part of rust-lang#102282. Opening it earlier so it's not mixed in with behavior changes and to avoid rebase hell. cc rust-lang#94829 (which nows has the hackmd linked). r? `@Mark-Simulacrum`
This comment was marked as resolved.
This comment was marked as resolved.
onur-ozkan
removed
E-mentor
Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change-tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Signed-off-by: onur-ozkan <[email protected]>
Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change-tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Signed-off-by: onur-ozkan <[email protected]>
Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change-tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Signed-off-by: onur-ozkan <[email protected]>
Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change-tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Signed-off-by: onur-ozkan <[email protected]>
… r=albertlarsan68 run change tracker even when config parse fails Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Fixes rust-lang#121756
Rollup merge of rust-lang#121787 - onur-ozkan:improve-change-tracker, r=albertlarsan68 run change tracker even when config parse fails Please note that we are currently validating the build configuration on two entry points (e.g., profile validation is handled on the python side), and change tracker system is handled on the rust side. Once rust-lang#94829 is completed (scheduled for 2024), we will be able to handle this more effectively. Fixes rust-lang#121756
Distribute bootstrap in CI This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here: 1. Make it faster to build rust from source, both the first time and incrementally 2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information. There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage. Helps with rust-lang/rust#94829.
Distribute bootstrap in CI This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here: 1. Make it faster to build rust from source, both the first time and incrementally 2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information. There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage. Helps with rust-lang/rust#94829.
This is a tracking issue for removing the requirement on python for bootstrapping the compiler.
Note that bootstrap.py is never going to be removed altogether, since distros still need to get a bootstrap compiler from somewhere; this issue is about contributors who already have a rust toolchain installed.
The living document for this tracking issue is https://hackmd.io/j-XXBeYERuajJknd6zErPA?both.
In particular, this subsumes #71818.
About tracking issues
Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
Steps
cargo run -p bootstrapdownload-llvmto rustbuild: Movedownload-ci-llvmout of bootstrap.py #95170. This can just shell out tocurl, the same way bootstrap.py does currently.download-rustcto rustbuild: Move download-rustc from python to rustbuild #96687cargo runwith bootstrap uses the host rustfmt, not nightly rustfmt #95136x.shandx.ps1entrypoints which don't go through bootstrap.py and download bootstrap from CIx.shandx.ps1shell scripts #99992bootstrap-shimbinary that only loads the config info and downloads bootstrap from CIx.sh/x.ps1src/tools/xto use the new entrypoints: Make the x tool use the x and x.ps1 scripts #105844If you'd like to help with this feature, feel free to join the Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/122652-new-members/topic/Help.20with.20RRIR.20bootstrap.20entrypoint.20.2394829
Unresolved Questions
Implementation history
cargo run -p bootstrap: Move some more bootstrap logic from python to rust #92260cargo runinstead ofcargo run -p bootstrap#94796cargo run -p bootstrap#94828cargo run tidy#94806download-ci-llvmout of bootstrap.py #95170cargo testfor bootstrap #95253cargo runon Windows #95254@rustbot label: +A-rustbuild +T-infra
cc @Mark-Simulacrum - I thought I'd write my plan down so you're not seeing PRs come in out of the blue.
The text was updated successfully, but these errors were encountered: