Deadlock with 0.18

[torokati44]

Description

We're getting regular deadlocks running some of our visual tests of Ruffle after upgrading to v0.18 (ruffle-rs/ruffle#13732), both on GitHub Actions, and our local machines.

Repro steps

$ git clone https://github.com/ruffle-rs/ruffle.git
$ cd ruffle
$ git checkout 375c99e601b28683058d09ad7ea76deb423ced1f
$ while true; do cargo test --all --features=imgtests stage3d -- --test-threads 16; done
# With 16 test threads, it's fairly quick; with 8 it still doesn't take long; with 4, it takes a while.
# Any single test running on its own, or any number of tests ran on a single thread, always work fine.

Expected vs observed behavior

Expected: No deadlock ever.
Observed: Eventual deadlock.

Extra materials

Attached are the parts of 8 deadlocked test runner threads' stack traces deemed relevant.

Thread 1

parking_lot::raw_rwlock::RawRwLock::lock_common (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115)
parking_lot::raw_rwlock::RawRwLock::lock_shared_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109)
lock_api::rwlock::RwLock<R,T>::read (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459)
wgpu_core::registry::Registry<T,I,F>::read (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:107)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::texture_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:868)
<wgpu::backend::direct::Context as wgpu::context::Context>::texture_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1633)
<T as wgpu::context::DynContext>::texture_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2576)
<wgpu::Texture as core::ops::drop::Drop>::drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:3360)
core::ptr::drop_in_place<wgpu::Texture> (Unknown Source:0)

Thread 2

parking_lot::raw_rwlock::RawRwLock::wait_for_readers (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1013)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:644)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:73)
lock_api::rwlock::RwLock<R,T>::write (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:491)
wgpu_core::registry::FutureId<I,T>::assign (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:64)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::device_create_bind_group (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:1354)
<wgpu::backend::direct::Context as wgpu::context::Context>::device_create_bind_group (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1101)
<T as wgpu::context::DynContext>::device_create_bind_group (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2308)
wgpu::Device::create_bind_group (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:2507)

Thread 3

parking_lot::raw_rwlock::RawRwLock::wait_for_readers (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1013)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:644)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:73)
lock_api::rwlock::RwLock<R,T>::write (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:491)
wgpu_core::registry::Registry<T,I,F>::write (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:132)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::buffer_unmap (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:3007)
<wgpu::backend::direct::Context as wgpu::context::Context>::buffer_unmap (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1563)
<T as wgpu::context::DynContext>::buffer_unmap (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2528)
wgpu::Buffer::unmap (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:3121)
<wgpu::Device as wgpu::util::device::DeviceExt>::create_buffer_init (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/util/device.rs:71)

Thread 4

parking_lot::raw_rwlock::RawRwLock::lock_common (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115)
parking_lot::raw_rwlock::RawRwLock::lock_shared_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109)
lock_api::rwlock::RwLock<R,T>::read (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459)
wgpu_core::registry::Registry<T,I,F>::read (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:107)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:155)
<wgpu::backend::direct::Context as wgpu::context::Context>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1288)
<T as wgpu::context::DynContext>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2359)
wgpu::Device::create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:2588)
<wgpu::Device as wgpu::util::device::DeviceExt>::create_buffer_init (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/util/device.rs:49)

Thread 5

parking_lot::raw_rwlock::RawRwLock::lock_common (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115)
parking_lot::raw_rwlock::RawRwLock::lock_shared_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109)
lock_api::rwlock::RwLock<R,T>::read (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459)
wgpu_core::registry::Registry<T,I,F>::read (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:107)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:155)
<wgpu::backend::direct::Context as wgpu::context::Context>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1288)
<T as wgpu::context::DynContext>::device_create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2359)
wgpu::Device::create_buffer (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:2588)

Thread 6

parking_lot::raw_rwlock::RawRwLock::wait_for_readers (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1013)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:644)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:73)
lock_api::rwlock::RwLock<R,T>::write (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:491)
wgpu_core::registry::Registry<T,I,F>::write (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:132)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::bind_group_layout_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:1189)
<wgpu::backend::direct::Context as wgpu::context::Context>::bind_group_layout_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1670)
<T as wgpu::context::DynContext>::bind_group_layout_drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2610)
<wgpu::BindGroupLayout as core::ops::drop::Drop>::drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:449)
core::ptr::drop_in_place<wgpu::BindGroupLayout> (Unknown Source:0)

Thread 7

parking_lot::raw_rwlock::RawRwLock::lock_common (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115)
parking_lot::raw_rwlock::RawRwLock::lock_shared_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109)
lock_api::rwlock::RwLock<R,T>::read (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459)
wgpu_core::registry::Registry<T,I,F>::read (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:107)
wgpu_core::device::global::<impl wgpu_core::global::Global<G>>::device_create_texture (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/device/global.rs:602)
<wgpu::backend::direct::Context as wgpu::context::Context>::device_create_texture (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1317)
<T as wgpu::context::DynContext>::device_create_texture (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2371)
wgpu::Device::create_texture (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:2605)

Thread 8

parking_lot::raw_rwlock::RawRwLock::lock_common (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115)
parking_lot::raw_rwlock::RawRwLock::lock_shared_slow (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109)
lock_api::rwlock::RwLock<R,T>::read (/home/attila/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459)
wgpu_core::registry::Registry<T,I,F>::read (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/registry.rs:107)
wgpu_core::command::render::<impl wgpu_core::global::Global<G>>::command_encoder_run_render_pass_impl (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/command/render.rs:1380)
wgpu_core::command::render::<impl wgpu_core::global::Global<G>>::command_encoder_run_render_pass (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu-core/src/command/render.rs:1303)
<wgpu::backend::direct::Context as wgpu::context::Context>::command_encoder_end_render_pass (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/backend/direct.rs:1981)
<T as wgpu::context::DynContext>::command_encoder_end_render_pass (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/context.rs:2830)
<wgpu::RenderPass as core::ops::drop::Drop>::drop (/home/attila/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/e16f7b4/wgpu/src/lib.rs:4219)
core::ptr::drop_in_place<wgpu::RenderPass> (Unknown Source:0)

Platform

Fedora Linux 39, Rust 1.74, Mesa 23.2.1, Ryzen 9 5950X, AMD Radeon RX 6600 XT (navi23, LLVM 16.0.6, DRM 3.54, 6.6.4-200.fc39.x86_64)
wgpu: branch v0.18, commit e16f7b4, mostly due to #4693, but also #4778.

[cwfitzgerald]

Could you try this again with trunk? We just fixed a few possible deadlocks - if it reproduces with the same stacks, I will investigate.

[torokati44]

That won't be easy - we're still on 0.18.x, and AFAIK there were some nontrivial API changes in trunk since then.
Could you perhaps backport the fixes to that branch?

[Dinnerbone]

It takes the ecosystem (specifically naga-oil and egui for us) a long while to update too, so even when trunk is out it may not be viable for us for a while :(

[torokati44]

Yeah, even the arcanization PR I was only able to test (around when it got merged) by ripping out egui entirely, and forking naga_oil.

[cwfitzgerald]

Not trivially unfortunately, as arcanization totally shook up wgpu-core internals and these issues very much depend on it.

[torokati44]

Well then, I suppose, back to waiting for 0.19, and for the ecosystem to update to it? 🙃

[torokati44]

I was able to test on current trunk with ruffle-rs/ruffle#14682, and the deadlock did not present itself at all even in a far longer run than for how long it took before to hang.
Thank you!

[cwfitzgerald]

Amazing news!