Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prerequisites from dep graph refactoring #2 #61923

Merged
merged 1 commit into from
Jul 11, 2019

Conversation

Zoxc
Copy link
Contributor

@Zoxc Zoxc commented Jun 18, 2019

Split out from #60035 and overlaps with #60559.

@rust-highfive
Copy link
Collaborator

r? @pnkfelix

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 18, 2019
@Centril
Copy link
Contributor

Centril commented Jun 18, 2019

cc @RalfJung

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:1c81285e:start=1560824301467688192,finish=1560824302229752957,duration=762064765
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:03:55] 
[01:03:55] running 9 tests
[01:03:55] iiiiiiiii
[01:03:55] 
[01:03:55]  finished in 0.146
[01:03:55] travis_fold:end:test_assembly


[01:03:55] travis_time:end:test_assembly:start=1560828147417072851,finish=1560828147563427265,duration=146354414

[01:03:55] travis_fold:start:test_incremental
travis_time:start:test_incremental
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:03:55] 
[01:03:55] running 103 tests
[01:04:09] ................................................................................F................... 100/103
[01:04:10] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:04:10] F..
[01:04:10] failures:
[01:04:10] 
[01:04:10] 
[01:04:10] ---- [incremental] incremental/rlib_cross_crate/b.rs stdout ----
[01:04:10] 
[01:04:10] error in revision `rpass3`: compilation failed!
[01:04:10] status: exit code: 101
[01:04:10] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/incremental/rlib_cross_crate/b.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--cfg" "rpass3" "-C" "incremental=/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/rlib_cross_crate/b/b.inc" "-Z" "incremental-verify-ich" "-Z" "incremental-queries" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/rlib_cross_crate/b/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/rlib_cross_crate/b/auxiliary"
[01:04:10] ------------------------------------------
[01:04:10] 
[01:04:10] ------------------------------------------
[01:04:10] stderr:
[01:04:10] stderr:
[01:04:10] ------------------------------------------
[01:04:10] warning: unused variable: `x`
[01:04:10]   --> /checkout/src/test/incremental/rlib_cross_crate/b.rs:25:9
[01:04:10]    |
[01:04:10] LL |     let x: a::Y = 'c';
[01:04:10]    |         ^ help: consider prefixing with an underscore: `_x`
[01:04:10]    = note: #[warn(unused_variables)] on by default
[01:04:10] 
[01:04:10] 
[01:04:10] thread 'rustc' panicked at 'DepGraph::try_mark_previous_green() - Duplicate DepNodeColor insertion for check_mod_liveness(b[317d])', src/librustc/dep_graph/graph.rs:783:9
[01:04:10] 
[01:04:10] error: internal compiler error: unexpected panic
[01:04:10] 
[01:04:10] note: the compiler unexpectedly panicked. this is a bug.
[01:04:10] note: the compiler unexpectedly panicked. this is a bug.
[01:04:10] 
[01:04:10] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:04:10] 
[01:04:10] note: rustc 1.37.0-dev running on x86_64-unknown-linux-gnu
[01:04:10] 
[01:04:10] note: compiler flags: -Z threads=1 -Z incremental-verify-ich -Z incremental-queries -Z ui-testing -Z unstable-options -Z query-dep-graph -C incremental -C rpath -C debuginfo=0
[01:04:10] 
[01:04:10] ------------------------------------------
[01:04:10] 
[01:04:10] 
[01:04:10] 
[01:04:10] ---- [incremental] incremental/type_alias_cross_crate/b.rs stdout ----
[01:04:10] 
[01:04:10] error in revision `rpass3`: compilation failed!
[01:04:10] status: exit code: 101
[01:04:10] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/incremental/type_alias_cross_crate/b.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--cfg" "rpass3" "-C" "incremental=/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/type_alias_cross_crate/b/b.inc" "-Z" "incremental-verify-ich" "-Z" "incremental-queries" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/type_alias_cross_crate/b/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Z" "query-dep-graph" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/type_alias_cross_crate/b/auxiliary"
[01:04:10] ------------------------------------------
[01:04:10] 
[01:04:10] ------------------------------------------
[01:04:10] stderr:
[01:04:10] stderr:
[01:04:10] ------------------------------------------
[01:04:10] warning: unused variable: `x`
[01:04:10]   --> /checkout/src/test/incremental/type_alias_cross_crate/b.rs:19:9
[01:04:10]    |
[01:04:10] LL |     let x: a::Y = 'c';
[01:04:10]    |         ^ help: consider prefixing with an underscore: `_x`
[01:04:10]    = note: #[warn(unused_variables)] on by default
[01:04:10] 
[01:04:10] 
[01:04:10] thread 'rustc' panicked at 'DepGraph::try_mark_previous_green() - Duplicate DepNodeColor insertion for check_mod_liveness(b[317d])', src/librustc/dep_graph/graph.rs:783:9
[01:04:10] 
[01:04:10] error: internal compiler error: unexpected panic
[01:04:10] 
[01:04:10] note: the compiler unexpectedly panicked. this is a bug.
[01:04:10] note: the compiler unexpectedly panicked. this is a bug.
[01:04:10] 
[01:04:10] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[01:04:10] 
[01:04:10] note: rustc 1.37.0-dev running on x86_64-unknown-linux-gnu
[01:04:10] 
[01:04:10] note: compiler flags: -Z threads=1 -Z incremental-verify-ich -Z incremental-queries -Z ui-testing -Z unstable-options -Z query-dep-graph -C incremental -C prefer-dynamic -C rpath -C debuginfo=0
[01:04:10] 
[01:04:10] ------------------------------------------
[01:04:10] 
[01:04:10] 
---
[01:04:10] test result: FAILED. 101 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:04:10] 
[01:04:10] 
[01:04:10] 
[01:04:10] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/incremental" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "incremental" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:04:10] 
[01:04:10] 
[01:04:10] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:04:10] Build completed unsuccessfully in 0:59:23
---
travis_time:end:136f92c0:start=1560828165083095209,finish=1560828165087875601,duration=4780392
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:244134f2
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0ea01e60
travis_time:start:0ea01e60
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:068c4839
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@RalfJung
Copy link
Member

@Centril anything specific you want me to look at? I am mostly unfamiliar with the code this is touching.

@Centril
Copy link
Contributor

Centril commented Jun 18, 2019

@RalfJung Pinged you for the AtomicCell thing :) (see #61923 (comment))

@RalfJung
Copy link
Member

RalfJung commented Jun 18, 2019

Ah. :) Well besides the problem around comparing Undef values in both compare_exchange and compare_and_swap, AtomicCell has two more problems that I know of:

  • It uses seq-locks for data that is too big for atomic accesses. This relies on the LLVM memory model, while Rust officially follows the C++ memory model. The difference is that in C++, read-write races are UB, whereas in LLVM they are well-defined but the read returns Undef. Add LLVM atomic memcpy intrinsics, expose in core/std #58599 might help here.
  • It relies on the validity invariant for integers to not rule out Undef. IOW, it relies on let x: u32 = mem::uninitialized() not being UB.

old
}
}

#[derive(Debug)]
pub struct Atomic<T: Copy>(Cell<T>);
Copy link
Member

@RalfJung RalfJung Jun 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between Atomic and AtomicCell? Neither of these types is described in the comment at the top of the file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, Atomic is just used here to define "fake" versions of AtomicBool and friends. Seems worth a comment that this is not actually public even though it is marked pub.
(It is not public in the sense that using it directly would lead to build failures in parallel rustc, if I understand correctly.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(There still is no such comment that I can see.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments here.

bors added a commit that referenced this pull request Jun 22, 2019
[WIP] Make dep node indices persistent between sessions

This makes marking dep nodes green faster (and lock free in the case with no diagnostics). This change is split out from #60035.

Unlike #60035 this makes loading the dep graph slower because it loads 2 copies of the dep graph, one immutable and one mutable.

Based on #61845, #61779 and #61923.
@Zoxc
Copy link
Contributor Author

Zoxc commented Jun 22, 2019

@bors try

@bors
Copy link
Contributor

bors commented Jun 22, 2019

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

  1. git checkout dep-stream-prefix-2 (switch to your branch)
  2. git fetch upstream master (retrieve the latest master)
  3. git rebase upstream/master -p (rebase on top of it)
  4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
  5. git push self dep-stream-prefix-2 --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message
warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

@Zoxc
Copy link
Contributor Author

Zoxc commented Jun 22, 2019

@bors try

@bors
Copy link
Contributor

bors commented Jun 22, 2019

⌛ Trying commit b76f9fe06f7e0d66f5df48b92d5fbc84e09c01b8 with merge 739ec977abef6de9ed58fcc2e0cc822138c7ceab...

@bors
Copy link
Contributor

bors commented Jun 22, 2019

💔 Test failed - checks-travis

@rust-highfive
Copy link
Collaborator

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:0d3a6aa8
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
Cloning into 'rust-lang/rust'...
travis_time:end:0d3a6aa8:start=1561235299045578076,finish=1561235304535555566,duration=5489977490
$ cd rust-lang/rust
$ git checkout -qf 739ec977abef6de9ed58fcc2e0cc822138c7ceab
fatal: reference is not a tree: 739ec977abef6de9ed58fcc2e0cc822138c7ceab
The command "git checkout -qf 739ec977abef6de9ed58fcc2e0cc822138c7ceab" failed and exited with 128 during .
Your build has been stopped.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 22, 2019
@Zoxc Zoxc mentioned this pull request Jul 1, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 1, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 3, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 3, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 3, 2019
Centril added a commit to Centril/rust that referenced this pull request Jul 3, 2019
@@ -1037,19 +1056,19 @@ impl CurrentDepGraph {
fingerprint: Fingerprint
) -> DepNodeIndex {
debug_assert!(!self.node_to_node_index.contains_key(&dep_node));
self.intern_node(dep_node, edges, fingerprint).0
self.intern_node(dep_node, edges, fingerprint)
}

fn intern_node(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment above the fn intern_node signature, indicating what the bool in the return type represents?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bool is removed, so no =P

@pnkfelix
Copy link
Member

pnkfelix commented Jul 10, 2019

This PR seems reasonable to me. r=me once my nits and question from review are addressed.

@Zoxc
Copy link
Contributor Author

Zoxc commented Jul 10, 2019

@bors r=pnkfelix

@bors
Copy link
Contributor

bors commented Jul 10, 2019

📌 Commit af339b1d3bc7ccb0e76d70536a2fa2ceef11a5d7 has been approved by pnkfelix

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 10, 2019
@Zoxc
Copy link
Contributor Author

Zoxc commented Jul 10, 2019

@bors r-

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 10, 2019
@Zoxc
Copy link
Contributor Author

Zoxc commented Jul 11, 2019

@bors r=pnkfelix

@bors
Copy link
Contributor

bors commented Jul 11, 2019

📌 Commit 29e7bfd has been approved by pnkfelix

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 11, 2019
@Zoxc
Copy link
Contributor Author

Zoxc commented Jul 11, 2019

I moved the AtomicCell addition to #62577.

Centril added a commit to Centril/rust that referenced this pull request Jul 11, 2019
Prerequisites from dep graph refactoring rust-lang#2

Split out from rust-lang#60035 and overlaps with rust-lang#60559.
bors added a commit that referenced this pull request Jul 11, 2019
Rollup of 7 pull requests

Successful merges:

 - #61665 (core: check for pointer equality when comparing Eq slices)
 - #61923 (Prerequisites from dep graph refactoring #2)
 - #62270 (Move async-await tests from run-pass to ui)
 - #62425 (filedesc: don't use ioctl(FIOCLEX) on Linux)
 - #62476 (Continue refactoring macro expansion and resolution)
 - #62519 (Regression test for HRTB bug (issue 30786).)
 - #62557 (Fix typo in libcore/intrinsics.rs)

Failed merges:

r? @ghost
@bors
Copy link
Contributor

bors commented Jul 11, 2019

⌛ Testing commit 29e7bfd with merge 6907005...

@bors bors merged commit 29e7bfd into rust-lang:master Jul 11, 2019
@Zoxc Zoxc deleted the dep-stream-prefix-2 branch July 11, 2019 15:44
Centril added a commit to Centril/rust that referenced this pull request Jul 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants