Use normal mutable borrows in matches

[matthewjasper]

ref mut borrows are currently two-phase with NLL enabled. This changes them to be proper mutable borrows. To accommodate this, first the position of fake borrows is changed:

[ 1. Pre-match ]
       |
[ (old create fake borrows) ]
[ 2. Discriminant testing -- check discriminants ] <-+
       |                                             |
       | (once a specific arm is chosen)             |
       |                                             |
[ (old read fake borrows) ]                          |
[ 3. Create "guard bindings" for arm ]               |
[ (create fake borrows) ]                            |
       |                                             |
[ 4. Execute guard code ]                            |
[ (read fake borrows) ] --(guard is false)-----------+
       |
       | (guard results in true)
       |
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

The following additional changes are made to accommodate ref mut bindings:

• We no longer create fake Shared borrows. These borrows are no longer needed for soundness, just to avoid some arguably strange cases.
• Shallow borrows no longer conflict with existing borrows, avoiding conflicting access between the guard borrow access and the ref mut borrow.

There is some further clean up done in this PR:

• Avoid the "later used here" note for Shallow borrows (since it's not relevant with the message provided)
• Make any use of a two-phase borrow activate it.
• Simplify the cleanup_post_borrowck passes into a single pass.

cc #56254

r? @nikomatsakis

[matthewjasper]

@bors try

[bors]

⌛ Trying commit 34f7a41485e00806e7b82ec5b439972a53033e9c with merge 26f8d250bb3c8a85e812cf663c730ef9e0df9315...

[bors]

☀️ Test successful - checks-travis
State: approved= try=True

[matthewjasper]

@rust-lang/infra please can this have a crater run?

[pietroalbini]

@craterbot run start=master#03acbd71c977cd63ce5f39ba9b6fe9ffd578785a end=try#26f8d250bb3c8a85e812cf663c730ef9e0df9315 mode=check-only

[craterbot]

👌 Experiment pr-57609 created and queued.
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🚧 Experiment pr-57609 is now running on agent aws-2.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-57609 is completed!
📊 0 regressed and 0 fixed (50551 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[matthewjasper]

A bit of a longer write up on the two options here then.

CFG construction changes

For reference, the generated CFG for a match with 3 arms where the first one has a guard now looks something like this:

The binding_blocks array is new (the blocks in it were mostly there already).

Fake borrows changes

Fake borrows are now added not only to the start block, but also to the end of any binding block that are proceeded by a guard. This means that the fake borrows are not live for the binding block and so do not conflict with any ref mut bindings.

Option A (commit 1)

At this point using minimal two-phase borrows is compatible with the match lowering. Some notes:

• This is the more minimal change to the current state
• This ties what's allowed in matches to what's allowed with 2 phase borrows
• Requires 2 + #patterns in the same arm different versions of the variable.
• Uses more fake borrows.

Option B (commit 3)

This uses real borrows. See OP for some details. Some note:

• This doesn't allow anything that the naive lowering doesn't.
• There are only 2 versions of each variable.
• The bound variable has a stable address.
• Requires changes to how the borrow checker handles fake borrows and fake reads
• Can cause some strange errors if a borrow from the guard conflicts with a borrow in the arm body.
• Potentially breaks code (although the crater run was clean).

[nikomatsakis]

@matthewjasper ok, I read your comment, but I still feel like I don't fully understand what is being proposed. Probably I'm just missing a certain amount of context. Let me just pepper out a few questions I guess:

Fake borrows are now added not only to the start block, but also to the end of any binding block that are proceeded by a guard.

When you say "fake borrows", you mean that we put dummy borrows like TMP = &foo, right? (Where the only uses of this TMP are artificial.) If I recall, we issue "shallow" borrows for basically all paths that lead to a discriminant that may be examined, more or less?

This means that the fake borrows are not live for the binding block...

Why does it mean that? Because the "fake borrows" we inserted at the start block never wind up being used? Are they live only during the matching code that we generate, which inspects the discriminants and so forth, or are they ever live for "user code" (i.e., guards)? If they are only live for code we generate, then it doesn't seem like they play much of a role, do they? (Since we trust that code not to mutate things.) Maybe they are needed for initialization checks or something?

At this point using minimal two-phase borrows is compatible with the match lowering.

What is a "minimal" 2PB? And which borrows are becoming 2PB exactly? Do you mean for the ref mut borrow that is used in the guard?

I feel like what I might like to see is some kind "representative desugaring" of a match in both styles.

[matthewjasper]

A simple example of the generated MIR for nightly and for this branch is here https://gist.github.com/matthewjasper/fb0f4bfb9d3a6d179ed0c2eb7a5b0da5 I've marked the 2-phase borrow that's created on nightly.

When you say "fake borrows", you mean that we put dummy borrows like TMP = &foo, right? (Where the only uses of this TMP are artificial.) If I recall, we issue "shallow" borrows for basically all paths that lead to a discriminant that may be examined, more or less?

Yes. In the example above, all of the shallow/guard borrows are "fake borrows". See bb7[4] and bb7[5] in the branch code for where they're being added.

Why does it mean that? Because the "fake borrows" we inserted at the start block never wind up being used? Are they live only during the matching code that we generate, which inspects the discriminants and so forth, or are they ever live for "user code" (i.e., guards)? If they are only live for code we generate, then it doesn't seem like they play much of a role, do they? (Since we trust that code not to mutate things.) Maybe they are needed for initialization checks or something?

They're live in all guards (and while we're branching). They're not live outside of the match expression, in the arm expressions or after we've selected a pattern and are creating bindings for it - either the bindings for the guard or the bindings for the arm.

What is a "minimal" 2PB? And which borrows are becoming 2PB exactly?

I mean the version of 2 phase borrows discussed in #56254. The two-phase borrows are the ones that are already two phase: the borrows associated to ref mut bindings for the guard. (e.g. the one at bb9[2] on nightly)

[nikomatsakis]

Sorry @matthewjasper been very distracted. I've added a calendar event to review and ponder this for Mon Jan 28, so hopefully that will help.

[nikomatsakis]

@matthewjasper

OK, I read the desugared example in your gist. Allow me to attempt to re-summarize what is going on.

First off, the match process looks vaguely like this:

[ 1. Pre-match ]
       |
[ 2. Discriminant testing -- check discriminants ] <-+
       | (once a specific arm is chosen)             |
[ 3. Create "guard bindings" for arm ]               |
       |                                             |
[ 4. Execute guard code ] --(guard is false)---------+
       | (guard results in true)
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

• We create a set of "fake borrows" for all discriminants that we may access
  at the start of the match, during stage 1. (In the gist, examples are the variables
  _8 and
  _9)
• We insert fake reads of those right before creating the guard bindings -- so at the start
  of stage 3 (e.g., in bb4 in the gist).
  • At this point, the "fake borrows" go out of scope, and we create the guard bindings
  • In the case of a ref mut binding, this involves a transient &mut creation
• Then we recreate the bindings at the end of stage 3
• We then re-read the fake borrows at the beginning of stage 5
  • this ensures that they are live during the guard code execution

Does this sound right?

[nikomatsakis]

To redraw my flow graph with fake borrows inserted, it looks something like this:

[ 1. Pre-match ]
[ (create fake borrows) ]
       |
[ 2. Discriminant testing -- check discriminants ] <-+
       |                                             |
       | (once a specific arm is chosen)             |
       |                                             |
[ (read fake borrows) ]                              |
[ 3. Create "guard bindings" for arm ]               |
[ (recreate fake borrows) ]                          |
       |                                             |
[ 4. Execute guard code ] --(guard is false)---------+
       |
       | (guard results in true)
       |
[ (read fake borrows) ]
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

[nikomatsakis]

If that is the case, I am wondering what the role of the fake borrows being live during "discriminant testing" is. I don't think they're harmful per se, just trying to create an accurate model of what the "threats" are.

I think there are a few things that users can do that are problematic:

• Use guard code to mutate something that is being matched
  • prevented by the fake borrows being live over the guard code
• Move things out that are being matched during the guard code
  • but that is prevented by desugaring x to the deref of a shared borrow

[matthewjasper]

So, first a couple of my own versions of the diagram:

[ 1. Pre-match ]
[ (create fake borrows) ] (*)
       |
[ 2. Discriminant testing -- check discriminants ] <-+
       |                                             |
       | (once a specific arm is chosen)             |
       |                                             |                            |
[ (read fake borrows) ]                              |
[ 3. Create "guard bindings" for arm (optional) ]    |
[ (recreate fake borrows ) ] (**)                    |
       |                                             |
[ 4. Execute guard code (opt) ] --(guard is false)---+
       |
       | (guard results in true)
       |
[ ( no fake read ) ] (+)
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

In the case of no guard

[ 1. Pre-match ]
[ (create fake borrows) ]
       |
[ 2. Discriminant testing -- check discriminants ]
       |
       | (once a specific arm is chosen) 
       |     
[ (read fake borrows) ] (***)
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

• We then re-read the fake borrows at the beginning of stage 5
  • this ensures that they are live during the guard code execution

No, there's no read here at the moment (you link to a fake read at (***)). It would certainly be nicer to have them there instead of where they currently are if it's not too annoying to implement. Currently we rely on the false edge to ensure that they are live during the guard.

If that is the case, I am wondering what the role of the fake borrows being live during "discriminant testing" is.

As well as the above I think you're asking why (*) exists, because that borrow is never live across user code. The reason is to (somewhat hackily) prevent a fake borrow from being live along a (**) -> [ Exit match ] -> (loop) -> [Pre-match] -> (***) path.

[matthewjasper]

So after some discussion on Zulip, I think that me and @nikomatsakis are agreed that we want the match CFG to look like this

[ 1. Pre-match ]
       |
[ 2. Discriminant testing -- check discriminants ] <-+
       |                                             |
       | (once a specific arm is chosen)             |
       |                                             |
[ 3. Create "guard bindings" for arm ]               |
[ (create fake borrows) ]                            |
       |                                             |
[ 4. Execute guard code ]                            |
[ (read fake borrows) ] --(guard is false)-----------+
       |
       | (guard results in true)
       |
[ 5. Create real bindings and execute arm ]
       |
[ Exit match ]

If there's no guard then 3. 4. and the fake borrows are omitted. This then gets us to the point where we can make 2-phase borrows conflict with all existing borrows, and possibly make the changes in the OP for being able to use real mutable borrows.

[Centril]

Too big to rollup, @bors p=1

[bors]

⌛ Testing commit 5ffc919 with merge 8972b60cf5f62fd21799a1609065a1eba92da65b...

[bors]

💔 Test failed - checks-travis

[rust-highfive]

The job x86_64-gnu-tools of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[01:54:00] normalized stderr:
[01:54:00] error[E0503]: cannot use `x` because it was mutably borrowed
[01:54:00]   --> $DIR/2phase.rs:49:35
[01:54:00]    |
[01:54:00] 49 |         ref mut y if { let _val = x; let _val = *y; true } => {},
[01:54:00]    |         |                         |
[01:54:00]    |         |                         use of borrowed `x`
[01:54:00]    |         borrow of `x` occurs here
[01:54:00] 
---
[01:54:00] 
[01:54:00] +error[E0503]: cannot use `x` because it was mutably borrowed
[01:54:00] +  --> $DIR/2phase.rs:49:35
[01:54:00] +   |
[01:54:00] +49 |         ref mut y if { let _val = x; let _val = *y; true } => {},
[01:54:00] +   |         |                         |
[01:54:00] +   |         |                         use of borrowed `x`
[01:54:00] +   |         borrow of `x` occurs here
[01:54:00] +
[01:54:00] +
[01:54:00] +error: aborting due to previous error
[01:54:00] +
[01:54:00] +For more information about this error, try `rustc --explain E0503`.
[01:54:00] +
[01:54:00] 
[01:54:00] The actual stderr differed from the expected stderr.
[01:54:00] Actual stderr saved to /tmp/compiletestmqsNl2/2phase.stderr
[01:54:00] To update references, run this command from build directory:
[01:54:00] tests/run-pass/update-references.sh '/tmp/compiletestmqsNl2' '2phase.rs'
[01:54:00] error: 1 errors occurred comparing output.
[01:54:00] status: exit code: 1
[01:54:00] status: exit code: 1
[01:54:00] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools-bin/miri" "tests/run-pass/2phase.rs" "-L" "/tmp/compiletestmqsNl2" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/tmp/compiletestmqsNl2/2phase.stage-id" "-Dwarnings" "-Dunused" "--edition" "2018" "-L" "/tmp/compiletestmqsNl2/2phase.stage-id.aux" "-A" "unused"
[01:54:00] ------------------------------------------
[01:54:00] 
[01:54:00] ------------------------------------------
[01:54:00] stderr:
[01:54:00] stderr:
[01:54:00] ------------------------------------------
[01:54:00] {"message":"cannot use `x` because it was mutably borrowed","code":{"code":"E0503","explanation":"\nA value was used after it was mutably borrowed.\n\nExample of erroneous code:\n\n```compile_fail,E0503\nfn main() {\n    let mut value = 3;\n    // Create a mutable borrow of `value`. This borrow\n    // lives until the end of this function.\n    let _borrow = &mut value;\n    let _sum = value + 1; // error: cannot use `value` because\n                          //        it was mutably borrowed\n}\n```\n\nIn this example, `value` is mutably borrowed by `borrow` and cannot be\nused to calculate `sum`. This is not possible because this would violate\nRust's mutability rules.\n\nYou can fix this error by limiting the scope of the borrow:\n\n```\nfn main() {\n    let mut value = 3;\n    // By creating a new block, you can limit the scope\n    // of the reference.\n    {\n        let _borrow = &mut value; // Use `_borrow` inside this block.\n    }\n    // The block has ended and with it the borrow.\n    // You can now use `value` again.\n    let _sum = value + 1;\n}\n```\n\nOr by cloning `value` before borrowing it:\n\n```\nfn main() {\n    let mut value = 3;\n    // We clone `value`, creating a copy.\n    let value_cloned = value.clone();\n    // The mutable borrow is a reference to `value` and\n    // not to `value_cloned`...\n    let _borrow = &mut value;\n    // ... which means we can still use `value_cloned`,\n    let _sum = value_cloned + 1;\n    // even though the borrow only ends here.\n}\n```\n\nYou can find more information about borrowing in the rust-book:\nhttp://doc.rust-lang.org/book/ch04-02-references-and-borrowing.html\n"},"level":"error","spans":[{"file_name":"tests/run-pass/2phase.rs","byte_start":687,"byte_end":696,"line_start":49,"line_end":49,"column_start":9,"column_end":18,"is_primary":false,"text":[{"text":"        ref mut y if { let _val = x; let _val = *y; true } => {},","highlight_start":9,"highlight_end":18}],"label":"borrow of `x` occurs here","suggested_replacement":null,"suggestion_applicability":null,"expansion":null},{"file_name":"tests/run-pass/2phase.rs","byte_start":713,"byte_end":714,"line_start":49,"line_end":49,"column_start":35,"column_end":36,"is_primary":true,"text":[{"text":"        ref mut y if { let _val = x; let _val = *y; true } => {},","highlight_start":35,"highlight_end":36}],"label":"use of borrowed `x`","suggested_replacement":null,"suggestion_applicability":null,"expansion":null},{"file_name":"tests/run-pass/2phase.rs","byte_start":727,"byte_end":729,"line_start":49,"line_end":49,"column_start":49,"column_end":51,"is_primary":false,"text":[{"text":"        ref mut y if { let _val = x; let _val = *y; true } => {},","highlight_start":49,"highlight_end":51}],"label":"borrow later used here","suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[],"rendered":"error[E0503]: cannot use `x` because it was mutably borrowed\n  --> tests/run-pass/2phase.rs:49:35\n   |\n49 |         ref mut y if { let _val = x; let _val = *y; true } => {},\n   |         ---------                 ^             -- borrow later used here\n   |         |                         |\n   |         |                         use of borrowed `x`\n   |         borrow of `x` occurs here\n\n"}
[01:54:00] {"message":"For more information about this error, try `rustc --explain E0503`.","code":null,"level":"","spans":[],"children":[],"rendered":"For more information about this error, try `rustc --explain E0503`.\n"}
[01:54:00] 
[01:54:00] ------------------------------------------
[01:54:00] 
---
travis_time:end:1030d175:start=1550967827236945108,finish=1550967827242958344,duration=6013236
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:191cdbbb
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:3ae7cc9e
travis_time:start:3ae7cc9e
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:108dcf73
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[kennytm]

Broken the miri test run-pass/2phase.rs, legit.

[matthewjasper]

@bors retry

[bors]

⌛ Testing commit 5ffc919 with merge 31eb0e2...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: pnkfelix
Pushing 31eb0e2 to master...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: pnkfelix
Pushing 31eb0e2 to master...