Fix drop-tracking ICE when a struct containing a field with a significant drop is used across an await #98754

jyn514 · 2022-07-01T05:25:12Z

Previously, drop-tracking would incorrectly assume the struct would be dropped immediately, which was not true.

Fixes #98476. Also fixes #98477, I think because the parent HIR node for type variables is the whole function instead of the expression where the variable is used.

r? @eholk

jyn514 · 2022-07-01T05:56:56Z

compiler/rustc_typeck/src/check/generator_interior.rs

+        let scope = if self.drop_ranges.is_borrowed_temporary(expr)
+            || ty.map_or(true, |ty| ty.has_significant_drop(self.fcx.tcx, self.fcx.param_env))


hmm. this is probably more coarse than necessary - it will also trigger for things like let x = String::new(); drop(x). I don't know a better way to do this - I guess this is what you were talking about in DMs?

eholk: But I think maybe the conservative thing to do is to count the field access (.nickname) as a borrow of the Config temporary, so we should add it to maybe_borrowed_temporaries. Then I think that should make the whole Config live for the whole temporary scope (until the semicolon). It's more conservative than it needs to be, since .nickname has been dropped but this would still consider it live since it's part of the type of Config, but it should be sound.

oh, it's still going to use the temporary scope though - I guess the proper test case is something like

f(String::new()).await

where the future doesn't actually use the string

oh lol while debugging this I found that there's another existing bug where we're too coarse: for non-Send types with trivial drop, we still consider them held across the generator:

async fn trivial_drop() { // uncomment to make this program compile // { let x: *const usize = &0; // } async {}.await; } fn assert_send<T: Send>(_: T) {} fn main() { assert_send(trivial_drop()); }

tried to fix this but got an ICE :( I think I will put this off until later

diff --git a/compiler/rustc_typeck/src/check/generator_interior.rs b/compiler/rustc_typeck/src/check/generator_interior.rs index 946bfa52d37..5caa1b2ded7 100644 --- a/compiler/rustc_typeck/src/check/generator_interior.rs +++ b/compiler/rustc_typeck/src/check/generator_interior.rs @@ -387,7 +387,10 @@ fn visit_expr(&mut self, expr: &'tcx Expr<'tcx>) { // temporary on the stack that is live for the current temporary scope and then return a // reference to it. That value may be live across the entire temporary scope. let scope = if self.drop_ranges.is_borrowed_temporary(expr) - || ty.map_or(true, |ty| ty.has_significant_drop(self.fcx.tcx, self.fcx.param_env)) + || ty.map_or(true, |ty| { + !ty.needs_drop(self.fcx.tcx, self.fcx.param_env) + || ty.has_significant_drop(self.fcx.tcx, self.fcx.param_env) + }) { self.rvalue_scopes.temporary_scope(self.region_scope_tree, expr.hir_id.local_id) } else {

thread 'rustc' panicked at 'index out of bounds: the len is 0 but the index is 13', /home/jnelson/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/ena-0.14.0/src/snapshot_vec.rs:199:10 21: 0x7f12f71e004b - ena::unify::UnificationTable<S>::value::h616d0c0c6ae4057b at /home/jnelson/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/ena-0.14.0/ src/unify/mod.rs:347:10 22: 0x7f12f71e004b - ena::unify::UnificationTable<S>::inlined_get_root_key::hea2bf8b4576b5472 at /home/jnelson/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/ena-0.14.0/ src/unify/mod.rs:362:19 23: 0x7f12f71e004b - ena::unify::UnificationTable<S>::inlined_probe_value::hc1dc88e31da4bcc9 at /home/jnelson/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/ena-0.14.0/ src/unify/mod.rs:566:18 24: 0x7f12f71e004b - rustc_infer::infer::type_variable::TypeVariableTable::inlined_probe::h29b4e108ffacbc08 at /home/jnelson/rust-lang/rust/compiler/rustc_infer/src/infer/type_variable.rs:312:9 25: 0x7f12f71e004b - rustc_infer::infer::type_variable::TypeVariableTable::probe::h91a8d70bb6dc3e56 at /home/jnelson/rust-lang/rust/compiler/rustc_infer/src/infer/type_variable.rs:306:9 26: 0x7f12f717fe34 - rustc_infer::infer::InferCtxt::probe_ty_var::hefc9501f755a1a65 at /home/jnelson/rust-lang/rust/compiler/rustc_infer/src/infer/mod.rs:1388:15 27: 0x7f12f71d7089 - <rustc_infer::infer::canonical::canonicalizer::Canonicalizer as rustc_middle::ty::fold::Ty peFolder>::fold_ty::h0ebb9c6ed6bc4918 at /home/jnelson/rust-lang/rust/compiler/rustc_infer/src/infer/canonical/canonicalizer .rs:394:23 ... query stack during panic: #0 [is_copy_raw] computing whether `[static generator@src/test/ui/async-await/non-trivial-drop-unused.rs:11:11: 11:13 ]` is `Copy` #1 [needs_drop_raw] computing whether `[static generator@src/test/ui/async-await/non-trivial-drop-unused.rs:11:11: 11 :13]` needs drop #2 [typeck] type-checking `trivial_drop` #3 [mir_built] building MIR for `trivial_drop` #4 [unsafety_check_result] unsafety-checking `trivial_drop` #5 [mir_const] processing MIR for `trivial_drop` #6 [mir_promoted] processing `trivial_drop` #7 [mir_borrowck] borrow-checking `trivial_drop` #8 [type_of] computing type of `trivial_drop::{opaque#0}` #9 [check_mod_item_types] checking item types in top-level module #10 [analysis] running analysis passes on this crate end of query stack

I guess the proper test case is something like

looks like this already errors today:

use std::rc::Rc; async fn ignore_param(x: Rc<i32>) { drop(x); async {}.await; } fn assert_send<T: Send>(_: T) {} fn main() { assert_send(ignore_param(Rc::new(0))); }

$ rustc +nightly src/test/ui/async-await/non-trivial-drop-unused.rs --edition 2018 -Zdrop-tracking error: future cannot be sent between threads safely --> src/test/ui/async-await/non-trivial-drop-unused.rs:19:17 | 19 | assert_send(ignore_param(Rc::new(0))); | ^^^^^^^^^^^^^^^^^^^^^^^^ future returned by `ignore_param` is not `Send` | = help: within `impl Future<Output = ()>`, the trait `Send` is not implemented for `Rc<i32>` note: captured value is not `Send` --> src/test/ui/async-await/non-trivial-drop-unused.rs:6:23 | 6 | async fn ignore_param(x: Rc<i32>) { | ^ has type `Rc<i32>` which is not `Send` note: required by a bound in `assert_send` --> src/test/ui/async-await/non-trivial-drop-unused.rs:16:19 | 16 | fn assert_send<T: Send>(_: T) {} | ^^^^ required by this bound in `assert_send`

~~that also seems wrong, for some reason it considers the param held for the whole async function even if you drop it earlier~~ this seems correct actually, the inner async block is Send but the outer one generated by the function desugaring is not. I tested with a non-async function and it compiles ok after this PR:

fn ignore_param(x: Rc<i32>) -> impl Future + Send { drop(x); async {} }

that's because we still need to normalize projections -- also, reminder that i said type variables, not type parameters. type variables don't have anything to do with param envs.

note a similar difference between Infcx::type_is_copy_modulo_regions and Ty::is_copy_modulo_regions.

Type (inference) variables are associated to a given InferCtxt, so unless the function is associated with one of those, it's not typically safe to pass a Ty to it unless you know that we're past the type-checking phase or we're outside of a function body.

thanks, using !self.fcx.inh.infcx.resolve_vars_if_possible(ty).needs_drop(self.fcx.tcx, self.fcx.param_env) fixed the ICE (but didn't allow the code to compile for reasons I don't understand - going to stop harping on it in this PR since it's not related to the current fix)

nit: fcx derefs to infcx so you should just be able to do fcx.resolve_vars_if_possible

also that'll still ice if you have any region variables, so you might need to pass it through erase_regions for types that have lifetimes and also significant drops

Ok, figured this out: #99105

jyn514 · 2022-07-02T06:13:45Z

... apparently this fixes #98477 too ?? I'll add a test.

eholk · 2022-07-06T22:21:20Z

This is awesome! I love it when it's just a small thing that was overlooked that fixes a bunch of different bugs.

(I don't know if this will work, but I'm going to try anyway...)
@bors r+

compiler-errors · 2022-07-06T22:26:05Z

hm, I'm not confident that this is the right solution...? Though I am only barely familiar with it.

eholk: But I think maybe the conservative thing to do is to count the field access (.nickname) as a borrow of the Config temporary, so we should add it to maybe_borrowed_temporaries. Then I think that should make the whole Config live for the whole temporary scope (until the semicolon). It's more conservative than it needs to be, since .nickname has been dropped but this would still consider it live since it's part of the type of Config, but it should be sound.

@jyn514, what happened to this convo that you mentioned above?

compiler-errors · 2022-07-06T22:29:54Z

but anyways if eholk is ok w/ the change then that's fine

@bors delegate=eholk

bors · 2022-07-06T22:29:56Z

✌️ @eholk can now approve this pull request

eholk · 2022-07-06T22:32:01Z

That was a conversation we had as a DM on Zulip (I guess in retrospect we probably should have discussed it in #wg-async).

This change seems reasonable to me. This is related to #94309, where we changed the scope we considered values live for to be smaller in some circumstances. Basically there the change was to use the parent expression as the scope, rather the temporary scope, unless the value was borrowed at some point (since the MIR counterpart to generator_interior is really conservative around borrows). It turns out that was a little too aggressive because if the value has a non-trivial drop, we need to consider it live until the destructor runs at the end of the temporary scope.

Let me know if that explanation makes sense. Some of the finer points of this gets pretty tricky and making sure I can explain it well helps me feel more confident in it.

compiler-errors · 2022-07-06T22:37:45Z

I think I understand. I guess the one part I am somewhat suspicious of is the usage of has_significant_drop over needs_drop -- the former seems RFC 2229 specific:

/// Note that this method is used to check for change in drop order for
/// 2229 drop reorder migration analysis.

eholk · 2022-07-06T22:46:32Z

Ah, good question. Let me read up a little more on that and figure out which one is more appropriate. (@jyn514 - I noticed at one point you used both? Do you know what the difference is?)

compiler-errors · 2022-07-06T22:48:19Z

I think we should be using needs_drop, but that one was ICEing due to some inference variable issues. I would rather us fix those inference variable issues properly.

If either of y'all need me to investigate how to get needs_drop to work without ICEing, lmk/ping me.

bors · 2022-07-08T06:27:43Z

☔ The latest upstream changes (presumably #98482) made this pull request unmergeable. Please resolve the merge conflicts.

jyn514 · 2022-07-10T01:31:43Z

I am not convinced we should be using needs_drop. Here is the documentation for both functions:
needs_drop():

If ty.needs_drop(...) returns true, then ty is definitely non-copy and might have a destructor attached; if it returns false, then ty definitely has no destructor (i.e., no drop glue).

has_significant_drop():

Checks if ty has has a significant drop.
Note that this method can return false even if ty has a destructor attached; even if that is the case then the adt has been marked with the attribute rustc_insignificant_dtor.

"non-copy" is not the right criteria here; we want this code (from #94309) to compile, even if Config contains types which aren't Copy:

fn status(_client_status: &Client) -> i16 {
    200
}

fn main() {
    let client = Client;
    let g = move || match status(&client) {
        _status => yield,
    };
    assert_send(g);
}

In general, I am very unsure why the change in #94309 is valid: it seems to me that any time Client has a drop impl, it's an observable change whether it's dropped before or after the yield. So the only time we should be using the parent expression instead of the end of the temporary expression is when the type is Copy.

figured this out: I was thinking about Client, not &Client. The example in that PR uses a struct which is !Sync, but still Send.

jyn514 · 2022-07-10T03:05:26Z

Ok, I pushed a test which shows why we need to use has_significant_drop instead of needs_drop. In particular, this variant of the test fails when using needs_drop: https://github.com/rust-lang/rust/blob/eaf6db8638a4a73c38fb7692876e406f3ba20014/src/test/ui/generator/drop-tracking-parent-expression.rs#L41-L42

jyn514 · 2022-07-10T20:17:36Z

I think I'm just going to do this for all non-copy types for now and worry about being more granular later. As a result the test case in #98754 (comment) will be broken until I get around to fixing this.

Ok, done. This also means the diagnostics for #97332 haven't been improved unfortunately.

@rustbot ready

compiler-errors

r=me with tiny nit applied and if @eholk is happy with these changes too

compiler/rustc_typeck/src/check/generator_interior.rs

…impl is used across an await Previously, drop-tracking would incorrectly assume the struct would be dropped immediately, which was not true: when the field had a type with a manual `Drop` impl, the drop becomes observable and has to be dropped after the await instead. For reasons I don't understand, this also fixes another error crater popped up related to type parameters. rust-lang#98476

eholk · 2022-07-11T23:46:52Z

Looks good to me!

Trying to be precise but not more precise than MIR has been a struggle with this feature :)

@bors r=compiler-errors

bors · 2022-07-11T23:46:54Z

📌 Commit b30315d has been approved by compiler-errors

It is now in the queue for this repository.

…-errors Fix drop-tracking ICE when a struct containing a field with a significant drop is used across an await Previously, drop-tracking would incorrectly assume the struct would be dropped immediately, which was not true. Fixes rust-lang#98476. Also fixes rust-lang#98477, I think because the parent HIR node for type variables is the whole function instead of the expression where the variable is used. r? `@eholk`

Dylan-DPC · 2022-07-12T11:35:48Z

likely failed in rollup

@bors r-

eholk · 2022-07-12T20:07:41Z

@bors rollup=never

jyn514 · 2022-07-13T22:55:57Z

I don't see how src/test/run-pass-valgrind/cast-enum-with-dtor.rs can be related. It uses no async code or generators, passes for me locally, and the error was because it got SIGKILL. I think it's much more likely that the MacOS runner just kills processes sometimes.

@bors r=compiler-errors

bors · 2022-07-13T22:55:59Z

📌 Commit b30315d has been approved by compiler-errors

It is now in the queue for this repository.

bors · 2022-07-14T02:22:51Z

⌛ Testing commit b30315d with merge 8a392a5...

bors · 2022-07-14T05:07:15Z

☀️ Test successful - checks-actions
Approved by: compiler-errors
Pushing 8a392a5 to master...

rust-timer · 2022-07-14T06:42:06Z

Finished benchmarking commit (8a392a5): comparison url.

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

Primary benchmarks: no relevant changes found
Secondary benchmarks: mixed results

	mean¹	max	count²
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	5.3%	5.3%	1
Improvements 🎉 (primary)	N/A	N/A	0
Improvements 🎉 (secondary)	-9.3%	-9.3%	1
All 😿🎉 (primary)	N/A	N/A	0

Cycles

Results

Primary benchmarks: 😿 relevant regression found
Secondary benchmarks: 😿 relevant regression found

	mean¹	max	count²
Regressions 😿 (primary)	2.3%	2.3%	1
Regressions 😿 (secondary)	3.1%	3.1%	1
Improvements 🎉 (primary)	N/A	N/A	0
Improvements 🎉 (secondary)	N/A	N/A	0
All 😿🎉 (primary)	2.3%	2.3%	1

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

the arithmetic mean of the percent change ↩ ↩²
number of relevant changes ↩ ↩²

jyn514 · 2022-07-14T12:21:33Z

The max-rss changes are all related to linking. Either the symbol names for generators have changed (very possible if we encode the types into the monomorphized name for some reason) or the benchmarks are noisy.

jyn514 added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-async-await Area: Async & Await labels Jul 1, 2022

jyn514 mentioned this pull request Jul 1, 2022

Fix printing impl trait under binders #98371

Merged

jyn514 commented Jul 1, 2022

View reviewed changes

jyn514 force-pushed the non-trivial-drop branch from 49a4a96 to eb2e322 Compare July 2, 2022 06:40

This comment has been minimized.

Sign in to view

jyn514 force-pushed the non-trivial-drop branch from eb2e322 to a7eace7 Compare July 2, 2022 06:57

eholk approved these changes Jul 6, 2022

View reviewed changes

This comment was marked as resolved.

Sign in to view

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 8, 2022

jyn514 added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 10, 2022

This comment was marked as outdated.

Sign in to view

jyn514 force-pushed the non-trivial-drop branch from a7eace7 to eaf6db8 Compare July 10, 2022 03:01

This comment was marked as resolved.

Sign in to view

compiler-errors approved these changes Jul 10, 2022

View reviewed changes

compiler/rustc_typeck/src/check/generator_interior.rs Show resolved Hide resolved

jyn514 force-pushed the non-trivial-drop branch from e5726c9 to 220abfa Compare July 10, 2022 20:31

jyn514 force-pushed the non-trivial-drop branch from 220abfa to b30315d Compare July 10, 2022 20:39

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 11, 2022

Dylan-DPC mentioned this pull request Jul 12, 2022

Rollup of 6 pull requests #99172

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 12, 2022

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 13, 2022

jyn514 assigned compiler-errors Jul 13, 2022

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 14, 2022

bors merged commit 8a392a5 into rust-lang:master Jul 14, 2022

rustbot added this to the 1.64.0 milestone Jul 14, 2022

jyn514 deleted the non-trivial-drop branch July 14, 2022 12:18

eholk mentioned this pull request Jul 14, 2022

Tracking issue for enabling -Zdrop-tracking by default #97331

Closed

7 tasks

		let scope = if self.drop_ranges.is_borrowed_temporary(expr)
		\|\| ty.map_or(true, \|ty\| ty.has_significant_drop(self.fcx.tcx, self.fcx.param_env))

Fix drop-tracking ICE when a struct containing a field with a significant drop is used across an await #98754

Fix drop-tracking ICE when a struct containing a field with a significant drop is used across an await #98754

Conversation

jyn514 commented Jul 1, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 Jul 1, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 Jul 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyn514 Jul 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyn514 commented Jul 2, 2022 • edited Loading

This comment has been minimized.

eholk commented Jul 6, 2022

This comment was marked as resolved.

This comment was marked as resolved.

compiler-errors commented Jul 6, 2022

compiler-errors commented Jul 6, 2022

bors commented Jul 6, 2022

eholk commented Jul 6, 2022 • edited Loading

compiler-errors commented Jul 6, 2022

eholk commented Jul 6, 2022

compiler-errors commented Jul 6, 2022

bors commented Jul 8, 2022

jyn514 commented Jul 10, 2022 • edited Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

jyn514 commented Jul 10, 2022 • edited Loading

This comment was marked as resolved.

jyn514 commented Jul 10, 2022

compiler-errors left a comment

Choose a reason for hiding this comment

eholk commented Jul 11, 2022

bors commented Jul 11, 2022

Dylan-DPC commented Jul 12, 2022

eholk commented Jul 12, 2022

jyn514 commented Jul 13, 2022 • edited Loading

bors commented Jul 13, 2022

bors commented Jul 14, 2022

bors commented Jul 14, 2022

rust-timer commented Jul 14, 2022

Footnotes

jyn514 commented Jul 14, 2022 • edited Loading

jyn514 commented Jul 1, 2022 •

edited

Loading

jyn514 Jul 1, 2022 •

edited

Loading

jyn514 Jul 1, 2022 •

edited

Loading

jyn514 Jul 1, 2022 •

edited

Loading

jyn514 commented Jul 2, 2022 •

edited

Loading

eholk commented Jul 6, 2022 •

edited

Loading

jyn514 commented Jul 10, 2022 •

edited

Loading

jyn514 commented Jul 10, 2022 •

edited

Loading

jyn514 commented Jul 13, 2022 •

edited

Loading

jyn514 commented Jul 14, 2022 •

edited

Loading