Add debug assertions to write_bytes and copy* #58783

nitnelave · 2019-02-27T16:15:12Z

This is a followup of #57997 to address #53871

We check that:

write_bytes has an aligned pointer
copy has both aligned pointers and the end of src isn't in dst (otherwise we overwrite something before reading it)
copy_nonoverlapping has both aligned pointers and src and dst don't overlap.

rust-highfive · 2019-02-27T16:15:18Z

r? @bluss

(rust_highfive has picked a reviewer for you, use r? to override)

nitnelave · 2019-02-27T16:15:54Z

r? @RalfJung

rust-highfive · 2019-02-27T16:30:42Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:272c64d4:start=1551284196264054014,finish=1551284197214879683,duration=950825669
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---

[00:03:52] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:52] tidy error: /checkout/src/libcore/intrinsics.rs:1514: line longer than 100 chars
[00:03:54] some tidy checks failed
[00:03:54] 
[00:03:54] 
[00:03:54] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:54] 
[00:03:54] 
[00:03:54] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:54] Build completed unsuccessfully in 0:00:48
[00:03:54] Build completed unsuccessfully in 0:00:48
[00:03:54] make: *** [tidy] Error 1
[00:03:54] Makefile:68: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:16491a40
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Wed Feb 27 16:20:43 UTC 2019
---
travis_time:end:009491f3:start=1551284444169764037,finish=1551284444175804930,duration=6040893
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:015ceb52
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:264ae4f9
travis_time:start:264ae4f9
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:09f4b64e
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive · 2019-02-27T17:23:29Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:000c9ac8:start=1551285321451041446,finish=1551285335930360595,duration=14479319149
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---
[00:47:37] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[00:47:37] 
[00:47:37] note: rustc 1.34.0-dev running on x86_64-unknown-linux-gnu
[00:47:37] 
[00:47:37] note: compiler flags: -Z force-unstable-if-unmarked -C opt-level=2 -C prefer-dynamic -C debug-assertions=y -C codegen-units=1 -C link-args=-Wl,-rpath,$ORIGIN/../lib --crate-type lib
[00:47:37] note: some of the compiler flags provided by cargo are hidden
[00:47:37] 
[00:47:37] error: Could not compile `core`.
[00:47:37]

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

nitnelave · 2019-02-27T17:31:26Z

Looks like something broke, we have a (questionably) invalid call to copy with src overlapping with dst. I'm investigating.

src/libcore/intrinsics.rs

RalfJung · 2019-02-27T18:00:50Z

src/libcore/intrinsics.rs

+        Ordering::Greater => dst_usize + size > src_usize,
+        // src == dst
+        Ordering::Equal => count != 0,
+    }


Uh, Miri will not like this... but yeah this seems right. Cc @oli-obk

RalfJung · 2019-02-27T18:02:20Z

Uh nice, maybe we found the first bug already? :D

nitnelave · 2019-02-27T18:02:50Z

While building the compiler, I got an error on a vector.clone() that triggered the overlapping part of copy-nonoverlapping. I'm having a look, but I'm not super familiar with the code, so it might take me a while.

RalfJung · 2019-02-27T18:04:08Z

Do you have a backtrace?

nitnelave · 2019-02-27T18:06:21Z

Yes, but I just switched off my computer, so I'll add it tomorrow.

…

On Wed, Feb 27, 2019, 19:04 Ralf Jung ***@***.***> wrote: Do you have a backtrace? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#58783 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwn2W5-CdEfn2au_y5xk_j5NQboOPZYks5vRsi4gaJpZM4bU5ug> .

src/libcore/intrinsics.rs

nitnelave · 2019-02-28T12:47:32Z

The stacktrace looks like this (trimmed to the relevant part):

10: core::slice::<impl [T]>::copy_from_slice         
             at ./src/libcore/macros.rs:11                                          
             at ./src/libcore/slice/mod.rs:1971                     
  11: <alloc::vec::Vec<T> as core::clone::Clone>::clone                          
             at ./src/liballoc/vec.rs:1915                                                                                                        
             at ./src/liballoc/vec.rs:1351                        
             at ./src/liballoc/slice.rs:160               
             at ./src/liballoc/slice.rs:380        
             at ./src/liballoc/vec.rs:1642                                                                                                           
 12: <rustc::traits::Vtable<'tcx, N> as core::clone::Clone>::clone             
             at src/librustc/traits/mod.rs:569                                  
             at src/librustc/traits/mod.rs:520                                  
  13: rustc::ty::erase_regions::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::erase_regions
             at src/librustc/ty/erase_regions.rs:26                                                 
  14: rustc::traits::codegen::<impl rustc::infer::InferCtxt<'a, 'gcx, 'tcx>>::drain_fulfillment_cx_or_panic
             at src/librustc/traits/codegen/mod.rs:161 
  15: rustc::ty::context::GlobalCtxt::enter_local                                                                                                               at src/librustc/traits/codegen/mod.rs:72                                                                                                           at src/librustc/infer/mod.rs:522                                                                  
             at src/librustc/ty/context.rs:1727                 
             at src/librustc/ty/context.rs:1966                                                                                                   
             at src/librustc/ty/context.rs:1899          
             at src/librustc/ty/context.rs:1965        
             at src/librustc/ty/context.rs:1726                                                                                                    
             at src/librustc/ty/context.rs:2072

RalfJung · 2019-02-28T14:35:13Z

10: core::slice::<impl [T]>::copy_from_slice

That function calls copy_nonoverlapping, so might be a real bug?

rust-highfive · 2019-02-28T15:21:38Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:30319caa:start=1551365512097169172,finish=1551365657099576484,duration=145002407312
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---
[00:27:02] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[00:27:02] 
[00:27:02] note: rustc 1.34.0-dev running on x86_64-unknown-linux-gnu
[00:27:02] 
[00:27:02] note: compiler flags: -Z force-unstable-if-unmarked -C opt-level=2 -C prefer-dynamic -C debug-assertions=y -C link-args=-Wl,-rpath,$ORIGIN/../lib --crate-type lib
[00:27:02] note: some of the compiler flags provided by cargo are hidden
[00:27:02] 
[00:27:02] error: Could not compile `rustc_apfloat`.
[00:27:02] warning: build failed, waiting for other jobs to finish...
---
145228 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu
145224 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release
142404 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps
141172 ./obj/build/bootstrap/debug/incremental/bootstrap-3i6jt5nchoxcn
141168 ./obj/build/bootstrap/debug/incremental/bootstrap-3i6jt5nchoxcn/s-f9wvmfulbl-lx0qp8-okq6jwyp562e
108528 ./src/llvm-project/lldb
97552 ./src/llvm-project/clang/test
89964 ./src/llvm-emscripten/test/CodeGen
78764 ./.git/modules
---
travis_time:end:010c1b2c:start=1551367295329126399,finish=1551367295333786765,duration=4660366
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:2042fa48
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
t

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Dylan-DPC-zz · 2019-03-18T13:59:26Z

ping from triage @nitnelave you have to fix the failing tests

RalfJung · 2019-03-18T15:33:28Z

The failure occurs in Vec::clone:

    fn clone(&self) -> Vec<T> {
        <[T]>::to_vec(&**self)
    }

I find it highly unlikely that there are indeed overlapping references here (and I think Miri would have found them), so my guess is that the check is wrong. OTOH, it looks all right... maybe try adding debug printing to overlaps?

If you do x.py --stage 0 test src/liballoc --no-doc, do you get test failures? That will build libstd but not the compiler, so you shouldn't run into the problem above.

RalfJung · 2019-03-18T15:36:31Z

src/libcore/intrinsics.rs

+    use crate::cmp::Ordering;
+    let src_usize = src as usize;
+    let dst_usize = dst as usize;
+    let size = mem::size_of::<T>() * count;


Could you use checked arithmetic here? Overflow should always panic.

Sure, doing that.

nitnelave · 2019-03-18T15:47:14Z

No, I don't get failures on x.py --stage 0 test src/liballoc. I'd love to blame it on the check, but I checked it again and can't see the issue :/

RalfJung · 2019-03-18T16:03:33Z

Well, then I guess you'll have to resort to printf debugging. :/

Let me know when you are stuck or need assistance.

ollie27 · 2019-03-18T16:11:02Z

I think the failure from #58783 (comment) was caused by the error I pointed out in #58783 (comment) because before that was fixed there was a false positive from slices of ZSTs. I think we need a new backtrace.

Co-Authored-By: nitnelave <[email protected]>

rust-highfive · 2019-04-27T05:54:33Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:040afaa4:start=1556344319110075927,finish=1556344319888331061,duration=778255134
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:01:56] extracting /checkout/obj/build/cache/2019-04-11/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:01:56] error: failed to resolve patches for `https://github.com/rust-lang/cargo`
[00:01:56] 
[00:01:56] Caused by:
[00:01:56]   patch for `cargo` in `https://github.com/rust-lang/cargo` did not resolve to any crates. If this is unexpected, you may wish to consult: https://github.com/rust-lang/cargo/issues/4678
[00:01:56] Build completed unsuccessfully in 0:00:14
[00:01:56] Makefile:69: recipe for target 'prepare' failed
[00:01:56] make: *** [prepare] Error 1
[00:01:57] Command failed. Attempt 2/5:
[00:01:57] Command failed. Attempt 2/5:
[00:01:57] error: failed to resolve patches for `https://github.com/rust-lang/cargo`
[00:01:57] 
[00:01:57] Caused by:
[00:01:57]   patch for `cargo` in `https://github.com/rust-lang/cargo` did not resolve to any crates. If this is unexpected, you may wish to consult: https://github.com/rust-lang/cargo/issues/4678
[00:01:57] Build completed unsuccessfully in 0:00:00
[00:01:57] make: *** [prepare] Error 1
[00:01:57] Makefile:69: recipe for target 'prepare' failed
[00:01:59] Command failed. Attempt 3/5:
[00:01:59] Command failed. Attempt 3/5:
[00:01:59] error: failed to resolve patches for `https://github.com/rust-lang/cargo`
[00:01:59] 
[00:01:59] Caused by:
[00:01:59]   patch for `cargo` in `https://github.com/rust-lang/cargo` did not resolve to any crates. If this is unexpected, you may wish to consult: https://github.com/rust-lang/cargo/issues/4678
[00:01:59] Build completed unsuccessfully in 0:00:00
[00:01:59] Makefile:69: recipe for target 'prepare' failed
[00:01:59] make: *** [prepare] Error 1
[00:02:02] Command failed. Attempt 4/5:
[00:02:02] Command failed. Attempt 4/5:
[00:02:02] error: failed to resolve patches for `https://github.com/rust-lang/cargo`
[00:02:02] 
[00:02:02] Caused by:
[00:02:02]   patch for `cargo` in `https://github.com/rust-lang/cargo` did not resolve to any crates. If this is unexpected, you may wish to consult: https://github.com/rust-lang/cargo/issues/4678
[00:02:02] Build completed unsuccessfully in 0:00:00
[00:02:02] make: *** [prepare] Error 1
[00:02:02] Makefile:69: recipe for target 'prepare' failed
[00:02:06] Command failed. Attempt 5/5:
[00:02:06] Command failed. Attempt 5/5:
[00:02:06] error: failed to resolve patches for `https://github.com/rust-lang/crates.io-index`
[00:02:06] 
[00:02:06] Caused by:
[00:02:06]   patch for `rustfmt-nightly` in `https://github.com/rust-lang/crates.io-index` did not resolve to any crates. If this is unexpected, you may wish to consult: https://github.com/rust-lang/cargo/issues/4678
[00:02:06] Build completed unsuccessfully in 0:00:00
[00:02:06] Makefile:69: recipe for target 'prepare' failed
[00:02:06] The command has failed after 5 attempts.
[00:02:06] make: *** [prepare] Error 1
---
travis_time:end:009498d2:start=1556344461575334217,finish=1556344461580946517,duration=5612300
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0a0c82bc
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:114d4d90
travis_time:start:114d4d90
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:047a23ba
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2019-04-29T20:51:27Z

☔ The latest upstream changes (presumably #60355) made this pull request unmergeable. Please resolve the merge conflicts.

RalfJung · 2019-04-30T06:48:41Z

@nitnelave what is the status of this? Something went very wrong in the last rebase, you now have tons of submodule changes in this PR.

nitnelave · 2019-04-30T07:42:55Z

Yes, I solved that but didn't push yet, I'm on holiday with little access to my computer :)

…

On Tue, Apr 30, 2019, 15:49 Ralf Jung ***@***.***> wrote: @nitnelave <https://github.com/nitnelave> what is the status of this? Something went very wrong in the last rebase, you now have tons of submodule changes in this PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#58783 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCPWILJWVC2EYZN427A2LPS7TYBANCNFSM4G2TTOQA> .

RalfJung · 2019-06-03T19:20:49Z

@nitnelave any updates on this?

Dylan-DPC-zz · 2019-06-24T13:28:31Z

ping from triage @nitnelave
Unfortunately we haven't heard from you on this in a while, so I'm closing the PR to keep things tidy. Don't worry though, if you'll have time again in the future please reopen this PR, we'll be happy to review it again! Thanks

RalfJung · 2019-06-24T21:20:27Z

Re-submitted as #62103.

nitnelave · 2019-06-25T07:08:53Z

Sorry about that, life got in the way :(

RalfJung · 2019-06-25T07:38:27Z

That's okay, and thanks for your work! I didn't have to do much any more. :)

@nitnelave