Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update Miri #105214

Merged
merged 32 commits into from
Dec 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
b6ae82b
replace 'locally built rustc' instructions by 'Miri in rustc'
RalfJung Nov 27, 2022
a0d104d
refactor scheduling of TLS dtors
RalfJung Nov 27, 2022
0849084
cleanup global imports a bit
RalfJung Nov 27, 2022
ec003fd
yield the main thread a number of times after its TLS dtors are done
RalfJung Nov 27, 2022
c9b9c17
add scoped thread test
RalfJung Nov 27, 2022
5238d17
fix TLS on partially supported OSes
RalfJung Nov 27, 2022
af92b04
move interpreter loop into thread manager; they are pretty tightly co…
RalfJung Nov 27, 2022
ad9784e
make ./miri run a bit more silent; add option to control seeds tested…
RalfJung Nov 28, 2022
63eae2b
add many-seeds capabilities to CI
RalfJung Nov 28, 2022
5c2592c
Auto merge of #2698 - RalfJung:miri-in-rustc, r=oli-obk
bors Nov 28, 2022
ef5d5e7
decreasw yield count a bit and explain reasoning a bit more
RalfJung Nov 28, 2022
623b4ab
Auto merge of #2699 - RalfJung:schedule-refactor, r=RalfJung
bors Dec 1, 2022
e8ff9b1
Bump ui_test crate
oli-obk Dec 1, 2022
4a12a13
Auto merge of #2707 - oli-obk:ui_test_bump, r=RalfJung
bors Dec 1, 2022
2d42d26
extract common borrow tracking logic
Vanille-N Dec 2, 2022
2528f4e
move stacked_borrows to borrow_tracker/stacked_borrows
Vanille-N Dec 2, 2022
3a01493
SbTag -> BorTag everywhere
Vanille-N Dec 2, 2022
8bb3d9e
other renames, introduction of BorrowTrackerMethod and AllocExtra
Vanille-N Dec 2, 2022
ab08f2a
fix imports
Vanille-N Dec 2, 2022
90118a1
Auto merge of #2697 - Vanille-N:borrow-tracking, r=RalfJung
bors Dec 2, 2022
5fd4b84
forward verbosity to cargo setup
RalfJung Dec 2, 2022
89dd322
Auto merge of #2708 - RalfJung:verbose-setup, r=RalfJung
bors Dec 2, 2022
595490e
slight simplifications for borrow tracking
RalfJung Dec 2, 2022
b12ce55
rename some more types for consistency
RalfJung Dec 2, 2022
0d1e365
fix ICE in pointer tracking
RalfJung Dec 2, 2022
4a64902
Auto merge of #2711 - RalfJung:btrack, r=RalfJung
bors Dec 2, 2022
7d75cc4
Auto merge of #2710 - RalfJung:ptr-tracking-ice, r=oli-obk
bors Dec 2, 2022
80ab672
Preparing for merge from rustc
RalfJung Dec 2, 2022
b64d867
Merge from rustc
RalfJung Dec 2, 2022
29814f2
clippy
RalfJung Dec 3, 2022
840f227
Auto merge of #2712 - RalfJung:rustup, r=RalfJung
bors Dec 3, 2022
229e65a
update lockfile
RalfJung Dec 3, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Cargo.lock
Original file line number Diff line number Diff line change
Expand Up @@ -5346,9 +5346,9 @@ checksum = "56dee185309b50d1f11bfedef0fe6d036842e3fb77413abef29f8f8d1c5d4c1c"

[[package]]
name = "ui_test"
version = "0.4.0"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf4559da3fe6b481f8674a29379677cb9606cd6f75fc254a2c9834c55638503d"
checksum = "54ddb6f31025943e2f9d59237f433711c461a43d9415974c3eb3a4902edc1c1f"
dependencies = [
"bstr 1.0.1",
"cargo_metadata 0.15.0",
Expand Down
61 changes: 14 additions & 47 deletions src/tools/miri/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,65 +203,32 @@ for more information about configuring VS Code and `rust-analyzer`.

[rdg-r-a]: https://rustc-dev-guide.rust-lang.org/building/suggested.html#configuring-rust-analyzer-for-rustc

## Advanced topic: other build environments
## Advanced topic: Working on Miri in the rustc tree

We described above the simplest way to get a working build environment for Miri,
which is to use the version of rustc indicated by `rustc-version`. But
sometimes, that is not enough.

### Building Miri with a locally built rustc
A big part of the Miri driver is shared with rustc, so working on Miri will
sometimes require also working on rustc itself. In this case, you should *not*
work in a clone of the Miri repository, but in a clone of the
[main Rust repository](https://github.com/rust-lang/rust/). There is a copy of
Miri located at `src/tools/miri` that you can work on directly. A maintainer
will eventually sync those changes back into this repository.

[building Miri with a locally built rustc]: #building-miri-with-a-locally-built-rustc
When working on Miri in the rustc tree, here's how you can run tests:

A big part of the Miri driver lives in rustc, so working on Miri will sometimes
require using a locally built rustc. The bug you want to fix may actually be on
the rustc side, or you just need to get more detailed trace of the execution
than what is possible with release builds -- in both cases, you should develop
Miri against a rustc you compiled yourself, with debug assertions (and hence
tracing) enabled.

The setup for a local rustc works as follows:
```sh
# Clone the rust-lang/rust repo.
git clone https://github.com/rust-lang/rust rustc
cd rustc
# Create a config.toml with defaults for working on Miri.
./x.py setup compiler
# Now edit `config.toml` and under `[rust]` set `debug-assertions = true`.

# Build a stage 2 rustc, and build the rustc libraries with that rustc.
# This step can take 30 minutes or more.
./x.py build --stage 2 compiler/rustc
# If you change something, you can get a faster rebuild by doing
./x.py build --keep-stage 0 --stage 2 compiler/rustc
# You may have to change the architecture in the next command
rustup toolchain link stage2 build/x86_64-unknown-linux-gnu/stage2
# Now cd to your Miri directory, then configure rustup
rustup override set stage2
```

Note: When you are working with a locally built rustc or any other toolchain that
is not the same as the one in `rust-version`, you should not have `.auto-everything` or
`.auto-toolchain` as that will keep resetting your toolchain.

```sh
rm -f .auto-everything .auto-toolchain
./x.py test miri --stage 0
```

Important: You need to delete the Miri cache when you change the stdlib; otherwise the
old, chached version will be used. On Linux, the cache is located at `~/.cache/miri`,
and on Windows, it is located at `%LOCALAPPDATA%\rust-lang\miri\cache`; the exact
location is printed after the library build: "A libstd for Miri is now available in ...".

Note: `./x.py --stage 2 compiler/rustc` currently errors with `thread 'main'
panicked at 'fs::read(stamp) failed with No such file or directory (os error 2)`,
you can simply ignore that error; Miri will build anyway.
`--bless` will work, too.

For more information about building and configuring a local compiler,
see <https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html>.
You can also directly run Miri on a Rust source file:

With this, you should now have a working development setup! See
[above](#building-and-testing-miri) for how to proceed working on Miri.
```
./x.py run miri --stage 0 --args src/tools/miri/tests/pass/hello.rs
```

## Advanced topic: Syncing with the rustc repo

Expand Down
4 changes: 2 additions & 2 deletions src/tools/miri/Cargo.lock
Original file line number Diff line number Diff line change
Expand Up @@ -724,9 +724,9 @@ dependencies = [

[[package]]
name = "ui_test"
version = "0.4.0"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf4559da3fe6b481f8674a29379677cb9606cd6f75fc254a2c9834c55638503d"
checksum = "54ddb6f31025943e2f9d59237f433711c461a43d9415974c3eb3a4902edc1c1f"
dependencies = [
"bstr",
"cargo_metadata",
Expand Down
2 changes: 1 addition & 1 deletion src/tools/miri/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ libloading = "0.7"

[dev-dependencies]
colored = "2"
ui_test = "0.4"
ui_test = "0.5"
rustc_version = "0.4"
# Features chosen to match those required by env_logger, to avoid rebuilds
regex = { version = "1.5.5", default-features = false, features = ["perf", "std"] }
Expand Down
5 changes: 2 additions & 3 deletions src/tools/miri/cargo-miri/src/phases.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ pub fn phase_cargo_miri(mut args: impl Iterator<Item = String>) {
let target = target.as_ref().unwrap_or(host);

// We always setup.
setup(&subcommand, target, &rustc_version);
setup(&subcommand, target, &rustc_version, verbose);

// Invoke actual cargo for the job, but with different flags.
// We re-use `cargo test` and `cargo run`, which makes target and binary handling very easy but
Expand Down Expand Up @@ -486,8 +486,7 @@ pub fn phase_runner(mut binary_args: impl Iterator<Item = String>, phase: Runner
continue;
} else if verbose > 0 {
eprintln!(
"[cargo-miri runner] Overwriting run-time env var {:?}={:?} with build-time value {:?}",
name, old_val, val
"[cargo-miri runner] Overwriting run-time env var {name:?}={old_val:?} with build-time value {val:?}"
);
}
}
Expand Down
19 changes: 12 additions & 7 deletions src/tools/miri/cargo-miri/src/setup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ use crate::util::*;
/// Performs the setup required to make `cargo miri` work: Getting a custom-built libstd. Then sets
/// `MIRI_SYSROOT`. Skipped if `MIRI_SYSROOT` is already set, in which case we expect the user has
/// done all this already.
pub fn setup(subcommand: &MiriCommand, target: &str, rustc_version: &VersionMeta) {
pub fn setup(subcommand: &MiriCommand, target: &str, rustc_version: &VersionMeta, verbose: usize) {
let only_setup = matches!(subcommand, MiriCommand::Setup);
let ask_user = !only_setup;
let print_sysroot = only_setup && has_arg_flag("--print-sysroot"); // whether we just print the sysroot path
Expand Down Expand Up @@ -99,12 +99,13 @@ pub fn setup(subcommand: &MiriCommand, target: &str, rustc_version: &VersionMeta
// `config.toml`.
command.env("RUSTC_WRAPPER", "");

if only_setup {
if print_sysroot {
// Be extra sure there is no noise on stdout.
command.stdout(process::Stdio::null());
if only_setup && !print_sysroot {
// Forward output. Even make it verbose, if requested.
for _ in 0..verbose {
command.arg("-v");
}
} else {
// Supress output.
command.stdout(process::Stdio::null());
command.stderr(process::Stdio::null());
}
Expand All @@ -120,7 +121,9 @@ pub fn setup(subcommand: &MiriCommand, target: &str, rustc_version: &VersionMeta
std::env::set_var("MIRI_SYSROOT", &sysroot_dir);

// Do the build.
if only_setup {
if print_sysroot {
// Be silent.
} else if only_setup {
// We want to be explicit.
eprintln!("Preparing a sysroot for Miri (target: {target})...");
} else {
Expand All @@ -143,7 +146,9 @@ pub fn setup(subcommand: &MiriCommand, target: &str, rustc_version: &VersionMeta
)
}
});
if only_setup {
if print_sysroot {
// Be silent.
} else if only_setup {
eprintln!("A sysroot for Miri is now available in `{}`.", sysroot_dir.display());
} else {
eprintln!("done");
Expand Down
11 changes: 8 additions & 3 deletions src/tools/miri/ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,15 @@ function run_tests {
./miri test
if [ -z "${MIRI_TEST_TARGET+exists}" ]; then
# Only for host architecture: tests with optimizations (`-O` is what cargo passes, but crank MIR
# optimizations up all the way).
# Optimizations change diagnostics (mostly backtraces), so we don't check them
#FIXME(#2155): we want to only run the pass and panic tests here, not the fail tests.
# optimizations up all the way, too).
# Optimizations change diagnostics (mostly backtraces), so we don't check
# them. Also error locations change so we don't run the failing tests.
MIRIFLAGS="${MIRIFLAGS:-} -O -Zmir-opt-level=4" MIRI_SKIP_UI_CHECKS=1 ./miri test -- tests/{pass,panic}

# Also run some many-seeds tests. 64 seeds means this takes around a minute per test.
for FILE in tests/many-seeds/*.rs; do
MIRI_SEEDS=64 CARGO_EXTRA_FLAGS="$CARGO_EXTRA_FLAGS -q" ./miri many-seeds ./miri run "$FILE"
done
fi

## test-cargo-miri
Expand Down
9 changes: 7 additions & 2 deletions src/tools/miri/miri
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ Mainly meant to be invoked by rust-analyzer.
./miri many-seeds <command>:
Runs <command> over and over again with different seeds for Miri. The MIRIFLAGS
variable is set to its original value appended with ` -Zmiri-seed=$SEED` for
many different seeds.
many different seeds. The MIRI_SEEDS variable controls how many seeds are being
tried; MIRI_SEED_START controls the first seed to try.

./miri bench <benches>:
Runs the benchmarks from bench-cargo-miri in hyperfine. hyperfine needs to be installed.
Expand Down Expand Up @@ -174,7 +175,9 @@ rustc-push)
fi
;;
many-seeds)
for SEED in $(seq 0 255); do
MIRI_SEED_START=${MIRI_SEED_START:-0} # default to 0
MIRI_SEEDS=${MIRI_SEEDS:-256} # default to 256
for SEED in $(seq $MIRI_SEED_START $(( $MIRI_SEED_START + $MIRI_SEEDS - 1 )) ); do
echo "Trying seed: $SEED"
MIRIFLAGS="$MIRIFLAGS -Zlayout-seed=$SEED -Zmiri-seed=$SEED" $@ || { echo "Failing seed: $SEED"; break; }
done
Expand Down Expand Up @@ -249,6 +252,8 @@ export RUSTFLAGS="-C link-args=-Wl,-rpath,$LIBDIR $RUSTFLAGS"
# Build a sysroot and set MIRI_SYSROOT to use it. Arguments are passed to `cargo miri setup`.
build_sysroot() {
if ! MIRI_SYSROOT="$($CARGO run $CARGO_EXTRA_FLAGS --manifest-path "$MIRIDIR"/cargo-miri/Cargo.toml -- miri setup --print-sysroot "$@")"; then
# Run it again so the user can see the error.
$CARGO run $CARGO_EXTRA_FLAGS --manifest-path "$MIRIDIR"/cargo-miri/Cargo.toml -- miri setup "$@"
echo "'cargo miri setup' failed"
exit 1
fi
Expand Down
2 changes: 1 addition & 1 deletion src/tools/miri/rust-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
454784afba5bf35b5ff14ada0e31265ad1d75e73
cef44f53034eac46be3a0e3eec7b2b3d4ef5140b
9 changes: 3 additions & 6 deletions src/tools/miri/src/bin/miri.rs
Original file line number Diff line number Diff line change
Expand Up @@ -192,10 +192,7 @@ fn init_late_loggers(tcx: TyCtxt<'_>) {
if log::Level::from_str(&var).is_ok() {
env::set_var(
"RUSTC_LOG",
format!(
"rustc_middle::mir::interpret={0},rustc_const_eval::interpret={0}",
var
),
format!("rustc_middle::mir::interpret={var},rustc_const_eval::interpret={var}"),
);
} else {
env::set_var("RUSTC_LOG", &var);
Expand Down Expand Up @@ -317,7 +314,7 @@ fn main() {
} else if arg == "-Zmiri-disable-validation" {
miri_config.validate = false;
} else if arg == "-Zmiri-disable-stacked-borrows" {
miri_config.stacked_borrows = false;
miri_config.borrow_tracker = None;
} else if arg == "-Zmiri-disable-data-race-detector" {
miri_config.data_race_detector = false;
miri_config.weak_memory_emulation = false;
Expand Down Expand Up @@ -413,7 +410,7 @@ fn main() {
err
),
};
for id in ids.into_iter().map(miri::SbTag::new) {
for id in ids.into_iter().map(miri::BorTag::new) {
if let Some(id) = id {
miri_config.tracked_pointer_tags.insert(id);
} else {
Expand Down
Loading