Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deadlock after updating to tracing-core 0.1.31 #2635

Closed
Noratrieb opened this issue Jun 27, 2023 · 8 comments
Closed

deadlock after updating to tracing-core 0.1.31 #2635

Noratrieb opened this issue Jun 27, 2023 · 8 comments
Labels
crate/core Related to the `tracing-core` crate kind/bug Something isn't working

Comments

@Noratrieb
Copy link

Version

I've narrowed the problem down to tracing-core 0.1.31, but here are all the tracing versions in tree after running cargo update, which reproduces the problem.

all the tracings 
    │   └── tracing v0.1.37
    │       ├── tracing-attributes v0.1.26 (proc-macro)
    │       └── tracing-core v0.1.31
├── tracing v0.1.37 (*)
├── tracing-subscriber v0.3.17
│   ├── tracing v0.1.37 (*)
│   ├── tracing-core v0.1.31 (*)
│   └── tracing-log v0.1.3
│       └── tracing-core v0.1.31 (*)
    │   │   ├── tracing-core v0.1.31 (*)
    │   │   └── tracing-error v0.2.0
    │   │       ├── tracing v0.1.37 (*)
    │   │       └── tracing-subscriber v0.3.17 (*)
    │   └── tracing-error v0.2.0 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   │   │   ├── tracing v0.1.37 (*)
│   │   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   │   │   │   ├── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   ├── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   │   │   ├── tracing v0.1.37 (*)
│       │   │   │   │   ├── tracing-subscriber v0.3.17 (*)
│       │   │   │   │   └── tracing-tree v0.2.3
│       │   │   │   │       ├── tracing-core v0.1.31 (*)
│       │   │   │   │       ├── tracing-log v0.1.3 (*)
│       │   │   │   │       └── tracing-subscriber v0.3.17 (*)
│       │   │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   ├── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   ├── tracing v0.1.37 (*)
│       │   ├── tracing-core v0.1.31 (*)
│       │   ├── tracing-subscriber v0.3.17 (*)
│       │   └── tracing-tree v0.2.3 (*)
│       └── tracing v0.1.37 (*)
    └── tracing v0.1.37 (*)
└── tracing v0.1.37 (*)
├── tracing v0.1.37 (*)
├── tracing-subscriber v0.3.17 (*)
└── tracing-tree v0.2.3 (*)
    │   └── tracing v0.1.37
    │       ├── tracing-attributes v0.1.26 (proc-macro)
    │       └── tracing-core v0.1.31
├── tracing v0.1.37 (*)
├── tracing-subscriber v0.3.17
│   ├── tracing v0.1.37 (*)
│   ├── tracing-core v0.1.31 (*)
│   └── tracing-log v0.1.3
│       └── tracing-core v0.1.31 (*)
    │   │   ├── tracing-core v0.1.31 (*)
    │   │   └── tracing-error v0.2.0
    │   │       ├── tracing v0.1.37 (*)
    │   │       └── tracing-subscriber v0.3.17 (*)
    │   └── tracing-error v0.2.0 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   │   │   ├── tracing v0.1.37 (*)
│   │   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   │   │   │   ├── tracing v0.1.37 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   │   ├── tracing v0.1.37 (*)
│   │   └── tracing v0.1.37 (*)
│   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   │   │   ├── tracing v0.1.37 (*)
│       │   │   │   │   ├── tracing-subscriber v0.3.17 (*)
│       │   │   │   │   └── tracing-tree v0.2.3
│       │   │   │   │       ├── tracing-core v0.1.31 (*)
│       │   │   │   │       ├── tracing-log v0.1.3 (*)
│       │   │   │   │       └── tracing-subscriber v0.3.17 (*)
│       │   │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   ├── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   │   └── tracing v0.1.37 (*)
│       │   └── tracing v0.1.37 (*)
│       │   ├── tracing v0.1.37 (*)
│       │   ├── tracing-core v0.1.31 (*)
│       │   ├── tracing-subscriber v0.3.17 (*)
│       │   └── tracing-tree v0.2.3 (*)
│       └── tracing v0.1.37 (*)
    └── tracing v0.1.37 (*)
└── tracing v0.1.37 (*)
├── tracing v0.1.37 (*)
├── tracing-subscriber v0.3.17 (*)
└── tracing-tree v0.2.3 (*)

This is after cargo update.

Platform

x86_64-unknown-linux-gnu in CI and aarch64-unknown-linux-gnu on my computer were both able to reproduce the deadlock, I haven't tried anything else.

Crates

tracing-core

Description

I updated all crates, which included tracing. After updating (which I've narrowed down to just bumping tracing-core from 0.1.30 to 0.1.31), the program (rustc) is stuck in a deadlock when debug logs are enabled. The deadlock happens after quite some logs are already emitted. rustc is single threaded, but all work happens on a second spawned thread, which deadlocks when locking a mutex.

gdb backtrace of that thread 
#0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1  0x0000ffffee86a2a4 in std::sys::unix::futex::futex_wait () at library/std/src/sys/unix/futex.rs:62
#2  0x0000ffffee879730 in std::sys::unix::locks::futex_mutex::Mutex::lock_contended () at library/std/src/sys/unix/locks/futex_mutex.rs:56
#3  0x0000fffff2229c4c in std::sys::unix::locks::futex_mutex::Mutex::lock () at library/std/src/sys/unix/locks/futex_mutex.rs:28
#4  std::sync::mutex::Mutex::lock<tracing_tree::format::Buffers> () at library/std/src/sync/mutex.rs:273
#5  tracing_tree::{impl#5}::on_event<tracing_subscriber::layer::layered::Layered<tracing_subscriber::filter::env::EnvFilter, tracing_subscriber::registry::sharded::Registry, tracing_subscriber::registry::sharded::Registry>, fn() -> std::io::stdio::Stderr, ()> () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-tree-0.2.3/src/lib.rs:324
#6  0x0000fffff7641e50 in tracing_core::event::Event::dispatch () at src/event.rs:34
#7  0x0000fffff740f818 in rustc_hir::definitions::{impl#4}::make::{closure#0}<rustc_hir::definitions::{impl#5}::def_path::{closure_env#0}> () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/macros.rs:664
#8  rustc_hir::definitions::DefPath::make<rustc_hir::definitions::{impl#5}::def_path::{closure_env#0}> () at compiler/rustc_hir/src/definitions.rs:200
#9  0x0000fffff7059438 in rustc_middle::ty::context::TyCtxt::def_path () at compiler/rustc_middle/src/ty/context.rs:823
#10 0x0000fffff7059f54 in rustc_middle::ty::context::TyCtxt::def_path_debug_str () at compiler/rustc_middle/src/ty/context.rs:898
#11 0x0000fffff222d5a8 in rustc_interface::callbacks::def_id_debug::{closure#0} () at compiler/rustc_interface/src/callbacks.rs:55
#12 rustc_middle::ty::context::tls::with_opt::{closure#0}<rustc_interface::callbacks::def_id_debug::{closure_env#0}, core::result::Result<(), core::fmt::Error>> () at compiler/rustc_middle/src/ty/context/tls.rs:154
#13 rustc_middle::ty::context::tls::with_context_opt<rustc_middle::ty::context::tls::with_opt::{closure_env#0}<rustc_interface::callbacks::def_id_debug::{closure_env#0}, core::result::Result<(), core::fmt::Error>>, core::result::Result<(), core::fmt::Error>> () at compiler/rustc_middle/src/ty/context/tls.rs:100
#14 rustc_middle::ty::context::tls::with_opt<rustc_interface::callbacks::def_id_debug::{closure_env#0}, core::result::Result<(), core::fmt::Error>> () at compiler/rustc_middle/src/ty/context/tls.rs:154
#15 rustc_interface::callbacks::def_id_debug () at compiler/rustc_interface/src/callbacks.rs:53
#16 0x0000ffffee8e8dc0 in core::fmt::builders::{impl#4}::field::{closure#0} () at library/core/src/fmt/builders.rs:322
#17 core::result::Result::and_then<(), core::fmt::Error, (), core::fmt::builders::{impl#4}::field::{closure_env#0}> () at library/core/src/result.rs:1319
#18 core::fmt::builders::DebugTuple::field () at library/core/src/fmt/builders.rs:309
#19 0x0000ffffee8ec204 in core::fmt::Formatter::debug_tuple_field2_finish () at library/core/src/fmt/mod.rs:2051
#20 0x0000fffff5887600 in rustc_hir::def::{impl#50}::fmt<rustc_ast::node_id::NodeId> () at compiler/rustc_hir/src/def.rs:307
#21 0x0000ffffee8e8dc0 in core::fmt::builders::{impl#4}::field::{closure#0} () at library/core/src/fmt/builders.rs:322
#22 core::result::Result::and_then<(), core::fmt::Error, (), core::fmt::builders::{impl#4}::field::{closure_env#0}> () at library/core/src/result.rs:1319
#23 core::fmt::builders::DebugTuple::field () at library/core/src/fmt/builders.rs:309
#24 0x0000ffffee8ec1a4 in core::fmt::Formatter::debug_tuple_field1_finish () at library/core/src/fmt/mod.rs:2035
#25 0x0000fffff572f5cc in core::option::{impl#50}::fmt<rustc_hir::def::Res<rustc_ast::node_id::NodeId>> () at library/core/src/option.rs:559
#26 0x0000ffffee8eaf0c in core::fmt::rt::Argument::fmt () at library/core/src/fmt/rt.rs:138
#27 core::fmt::write () at library/core/src/fmt/mod.rs:1094
#28 0x0000ffffee8ebc94 in core::fmt::Formatter::write_fmt () at library/core/src/fmt/mod.rs:1560
#29 0x0000fffff57519c8 in rustc_resolve::{impl#10}::fmt () at compiler/rustc_resolve/src/lib.rs:654
#30 0x0000ffffee8eaf0c in core::fmt::rt::Argument::fmt () at library/core/src/fmt/rt.rs:138
#31 core::fmt::write () at library/core/src/fmt/mod.rs:1094
#32 0x0000ffffee8eaf0c in core::fmt::rt::Argument::fmt () at library/core/src/fmt/rt.rs:138
#33 core::fmt::write () at library/core/src/fmt/mod.rs:1094
#34 0x0000fffff5eb7ef8 in core::fmt::Write::write_fmt<alloc::string::String> () at /home/gh-Nilstrieb/rust/library/core/src/fmt/mod.rs:192
#35 tracing_tree::format::{impl#3}::record_debug () at /home/gh-Nilstrieb/rust/library/core/src/macros/mod.rs:519
#36 0x0000fffff76446c4 in tracing_core::field::{impl#16}::record<dyn tracing_core::field::Value> () at src/field.rs:586
#37 tracing_core::field::ValueSet::record () at src/field.rs:1001
#38 0x0000fffff2229f30 in tracing_core::event::Event::record () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.31/src/event.rs:87
#39 tracing_tree::{impl#5}::on_event<tracing_subscriber::layer::layered::Layered<tracing_subscriber::filter::env::EnvFilter, tracing_subscriber::registry::sharded::Registry, tracing_subscriber::registry::sharded::Registry>, fn() -> std::io::stdio::Stderr, ()> () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-tree-0.2.3/src/lib.rs:402
#40 0x0000fffff7641e50 in tracing_core::event::Event::dispatch () at src/event.rs:34
#41 0x0000fffff5757360 in rustc_resolve::{impl#18}::resolve_crate_root::{closure#10} () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/macros.rs:664
#42 rustc_resolve::Resolver::resolve_crate_root () at compiler/rustc_resolve/src/lib.rs:1797
#43 0x0000fffff5881ff0 in rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure#0} () at compiler/rustc_resolve/src/macros.rs:182
#44 rustc_span::hygiene::update_dollar_crate_names::{closure#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}> () at compiler/rustc_span/src/hygiene.rs:620
#45 core::iter::adapters::map::map_fold::{closure#0}<usize, rustc_span::symbol::Symbol, (), rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>, core::iter::traits::iterator::Iterator::for_each::call::{closure_env#0}<rustc_span::symbol::Symbol, alloc::vec::{impl#19}::extend_trusted::{closure_env#0}<rustc_span::symbol::Symbol, alloc::alloc::Global, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>>>> () at library/core/src/iter/adapters/map.rs:84
#46 core::iter::traits::iterator::Iterator::fold<core::ops::range::Range<usize>, (), core::iter::adapters::map::map_fold::{closure_env#0}<usize, rustc_span::symbol::Symbol, (), rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>, core::iter::traits::iterator::Iterator::for_each::call::{closure_env#0}<rustc_span::symbol::Symbol, alloc::vec::{impl#19}::extend_trusted::{closure_env#0}<rustc_span::symbol::Symbol, alloc::alloc::Global, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>>>>> () at library/core/src/iter/traits/iterator.rs:2481
#47 core::iter::adapters::map::{impl#2}::fold<rustc_span::symbol::Symbol, core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>, (), core::iter::traits::iterator::Iterator::for_each::call::{closu
#48 0x0000fffff57cfcf8 in core::iter::traits::iterator::Iterator::for_each<core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>, alloc::vec::{impl#19}::extend_trusted::{closure_env#0}<rustc_span::symbol::Symbol, alloc::alloc::Global, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>>> () at library/core/src/iter/traits/iterator.rs:856
#49 alloc::vec::Vec::extend_trusted<rustc_span::symbol::Symbol, alloc::alloc::Global, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>> () at library/alloc/src/vec/mod.rs:2827
#50 alloc::vec::spec_extend::{impl#1}::spec_extend<rustc_span::symbol::Symbol, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>, alloc::alloc::Global> () at library/alloc/src/vec/spec_extend.rs:26
#51 alloc::vec::spec_from_iter_nested::{impl#1}::from_iter<rustc_span::symbol::Symbol, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>> () at library/alloc/src/vec/spec_from_iter_nested.rs:62
#52 alloc::vec::spec_from_iter::{impl#0}::from_iter<rustc_span::symbol::Symbol, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>> () at library/alloc/src/vec/spec_from_iter.rs:33
#53 0x0000fffff587f428 in alloc::vec::{impl#14}::from_iter<rustc_span::symbol::Symbol, core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>> () at library/alloc/src/vec/mod.rs:2695
#54 core::iter::traits::iterator::Iterator::collect<core::iter::adapters::map::Map<core::ops::range::Range<usize>, rustc_span::hygiene::update_dollar_crate_names::{closure_env#1}<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}>>, alloc::vec::Vec<rustc_span::symbol::Symbol, alloc::alloc::Global>> () at library/core/src/iter/traits/iterator.rs:1895
#55 rustc_span::hygiene::update_dollar_crate_names<rustc_resolve::macros::{impl#0}::resolve_dollar_crates::{closure_env#0}> () at compiler/rustc_span/src/hygiene.rs:620
#56 0x0000fffff68a80f0 in rustc_expand::expand::MacroExpander::collect_invocations () at compiler/rustc_expand/src/expand.rs:567
#57 0x0000fffff68a5ad4 in rustc_expand::expand::MacroExpander::fully_expand_fragment () at compiler/rustc_expand/src/expand.rs:415
#58 0x0000fffff68a5784 in rustc_expand::expand::MacroExpander::expand_crate () at compiler/rustc_expand/src/expand.rs:402
#59 0x0000fffff2275284 in rustc_interface::passes::configure_and_expand::{closure#1}::{closure#1} () at compiler/rustc_interface/src/passes.rs:244
#60 rustc_data_structures::profiling::VerboseTimingGuard::run<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure#1}::{closure_env#1}> () at compiler/rustc_data_structures/src/profiling.rs:752
#61 rustc_session::session::Session::time<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure#1}::{closure_env#1}> () at compiler/rustc_session/src/utils.rs:11
#62 rustc_interface::passes::configure_and_expand::{closure#1} () at compiler/rustc_interface/src/passes.rs:244
#63 rustc_data_structures::profiling::VerboseTimingGuard::run<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure_env#1}> () at compiler/rustc_data_structures/src/profiling.rs:752
#64 rustc_session::session::Session::time<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure_env#1}> () at compiler/rustc_session/src/utils.rs:11
#65 0x0000fffff22ea9cc in rustc_interface::passes::configure_and_expand () at compiler/rustc_interface/src/passes.rs:195
#66 rustc_interface::passes::resolver_for_lowering () at compiler/rustc_interface/src/passes.rs:591
#67 0x0000fffff60ceed4 in rustc_query_impl::query_impl::resolver_for_lowering::dynamic_query::{closure#2}::{closure#0} () at compiler/rustc_query_impl/src/plumbing.rs:582
#68 rustc_query_impl::plumbing::__rust_begin_short_backtrace<rustc_query_impl::query_impl::resolver_for_lowering::dynamic_query::{closure#2}::{closure_env#0}, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_query_impl/src/plumbing.rs:506
#69 0x0000fffff6126964 in rustc_query_impl::query_impl::resolver_for_lowering::dynamic_query::{closure#2} () at compiler/rustc_query_impl/src/plumbing.rs:578
#70 core::ops::function::FnOnce::call_once<rustc_query_impl::query_impl::resolver_for_lowering::dynamic_query::{closure_env#2}, (rustc_middle::ty::context::TyCtxt, ())> () at library/core/src/ops/function.rs:250
#71 0x0000fffff62fd380 in rustc_query_impl::{impl#2}::compute<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false> () at compiler/rustc_query_impl/src/lib.rs:116
#72 rustc_query_system::query::plumbing::execute_job_non_incr::{closure#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt> () at compiler/rustc_query_system/src/query/plumbing.rs:487
#73 rustc_middle::ty::context::tls::enter_context::{closure#0}<rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:82
#74 std::thread::local::LocalKey::try_with<core::cell::Cell<*const ()>, rustc_middle::ty::context::tls::enter_context::{closure_env#0}<rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at library/std/src/thread/local.rs:270
#75 std::thread::local::LocalKey::with<core::cell::Cell<*const ()>, rustc_middle::ty::context::tls::enter_context::{closure_env#0}<rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at library/std/src/thread/local.rs:246
#76 0x0000fffff615a0e8 in rustc_middle::ty::context::tls::enter_context<rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:79
#77 rustc_query_impl::plumbing::{impl#3}::start_query::{closure#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>
#78 rustc_middle::ty::context::tls::with_related_context::{closure#0}<rustc_query_impl::plumbing::{impl#3}::start_query::{closure_env#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:133
#79 rustc_middle::ty::context::tls::with_context::{closure#0}<rustc_middle::ty::context::tls::with_related_context::{closure_env#0}<rustc_query_impl::plumbing::{impl#3}::start_query::{closure_env#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:111
#80 rustc_middle::ty::context::tls::with_context_opt<rustc_middle::ty::context::tls::with_context::{closure_env#0}<rustc_middle::ty::context::tls::with_related_context::{closure_env#0}<rustc_query_impl::plumbing::{impl#3}::start_query::{closure_env#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:100
#81 rustc_middle::ty::context::tls::with_context<rustc_middle::ty::context::tls::with_related_context::{closure_env#0}<rustc_query_impl::plumbing::{impl#3}::start_query::{closure_env#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>>, rustc_middle::query::erase::Erased<[u8; 8]>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:111
#82 rustc_middle::ty::context::tls::with_related_context<rustc_query_impl::plumbing::{impl#3}::start_query::{closure_env#0}<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>>, rustc_middle::query::erase::Erased<[u8; 8]>> () at compiler/rustc_middle/src/ty/context/tls.rs:124
#83 rustc_query_impl::plumbing::{impl#3}::start_query<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::execute_job_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>> () at compiler/rustc_query_impl/src/plumbing.rs:134
#84 rustc_query_system::query::plumbing::execute_job_non_incr<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt> () at compiler/rustc_query_system/src/query/plumbing.rs:487
#85 rustc_query_system::query::plumbing::execute_job<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () at compiler/rustc_query_system/src/query/plumbing.rs:420
#86 rustc_query_system::query::plumbing::try_execute_query<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () at compiler/rustc_query_system/src/query/plumbing.rs:365
#87 0x0000fffff6294b20 in rustc_query_system::query::plumbing::get_query_non_incr::{closure#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt> () at compiler/rustc_query_sy
#88 stacker::maybe_grow<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::get_query_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>> () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/stacker-0.1.15/src/lib.rs:55
#89 rustc_data_structures::stack::ensure_sufficient_stack<rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::query::plumbing::get_query_non_incr::{closure_env#0}<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>> () at compiler/rustc_data_structures/src/stack.rs:17
#90 rustc_query_system::query::plumbing::get_query_non_incr<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt> () at compiler/rustc_query_system/src/query/plumbing.rs:817
#91 rustc_query_impl::query_impl::resolver_for_lowering::get_query_non_incr::__rust_end_short_backtrace () at compiler/rustc_query_impl/src/plumbing.rs:556
#92 0x0000fffff2204068 in rustc_middle::query::plumbing::query_get_at<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 8]>>> () at compiler/rustc_middle/src/query/plumbing.rs:155
#93 0x0000fffff2202df0 in rustc_middle::query::plumbing::TyCtxtAt::resolver_for_lowering () at compiler/rustc_middle/src/query/plumbing.rs:385
#94 rustc_middle::ty::context::TyCtxt::resolver_for_lowering () at compiler/rustc_middle/src/query/plumbing.rs:376
#95 rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure#2} () at compiler/rustc_driver_impl/src/lib.rs:402
#96 rustc_middle::ty::context::{impl#9}::enter::{closure#0}<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at compiler/rustc_middle/src/ty/context.rs:574
#97 rustc_middle::ty::context::tls::enter_context::{closure#0}<rustc_middle::ty::context::{impl#9}::enter::{closure_env#0}<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at compiler/rustc_middle/src/ty/context/tls.rs:82
#98 std::thread::local::LocalKey::try_with<core::cell::Cell<*const ()>, rustc_middle::ty::context::tls::enter_context::{closure_env#0}<rustc_middle::ty::context::{impl#9}::enter::{closure_env#0}<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at library/std/src/thread/local.rs:270
#99 std::thread::local::LocalKey::with<core::cell::Cell<*const ()>, rustc_middle::ty::context::tls::enter_context::{closure_env#0}<rustc_middle::ty::context::{impl#9}::enter::{closure_env#0}<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at library/std/src/thread/local.rs:246
#100 0x0000fffff2202c04 in rustc_middle::ty::context::tls::enter_context<rustc_middle::ty::context::{impl#9}::enter::{closure_env#0}<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>>, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at compiler/rustc_middle/src/ty/context/tls.rs:79
#101 rustc_middle::ty::context::GlobalCtxt::enter<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure_env#2}, &rustc_data_structures::steal::Steal<(rustc_middle::ty::ResolverAstLowering, alloc::rc::Rc<rustc_ast::ast::Crate>)>> () at compiler/rustc_middle/src/ty/context.rs:574
#102 0x0000fffff21d9adc in rustc_driver_impl::run_compiler::{closure#1}::{closure#2} () at compiler/rustc_driver_impl/src/lib.rs:402
#103 rustc_interface::interface::Compiler::enter<rustc_driver_impl::run_compiler::{closure#1}::{closure_env#2}, core::result::Result<core::option::Option<rustc_interface::queries::Linker>, rustc_span::ErrorGuaranteed>> () at compiler/rustc_interface/src/queries.rs:389
#104 0x0000fffff2211710 in rustc_driver_impl::run_compiler::{closure#1} () at compiler/rustc_driver_impl/src/lib.rs:363
#105 rustc_interface::interface::run_compiler::{closure#0}::{closure#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}> () at compiler/rustc_interface/src/interface.rs:332
#106 rustc_span::set_source_map<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler::{closure#0}::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>> () at compiler/rustc_span/src/lib.rs:1037
#107 0x0000fffff21eca84 in rustc_interface::interface::run_compiler::{closure#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}> () at compiler/rustc_interface/src/interface.rs:326
#108 scoped_tls::ScopedKey::set<rustc_span::SessionGlobals, rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at /home/gh-Nilstrieb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scoped-tls-1.0.1/src/lib.rs:137
#109 0x0000fffff22110bc in rustc_span::create_session_globals_then<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>> () at compiler/rustc_span/src/lib.rs:120
#110 rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at compiler/rustc_interface/src/util.rs:155
#111 std::sys_common::backtrace::__rust_begin_short_backtrace<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/sys_common/backtrace.rs:135
#112 0x0000fffff21c4b10 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/thread/mod.rs:529
#113 core::panic::unwind_safe::{impl#25}::call_once<core::result::Result<(), rustc_span::ErrorGuaranteed>, std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>> () at library/core/src/panic/unwind_safe.rs:271
#114 0x0000fffff21c4348 in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuara
#112 0x0000fffff21c4b10 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/thread/mod.rs:529
#113 core::panic::unwind_safe::{impl#25}::call_once<core::result::Result<(), rustc_span::ErrorGuaranteed>, std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>> () at library/core/src/panic/unwind_safe.rs:271
#114 0x0000fffff21c4348 in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuara--Type <RET> for more, q to quit, c to continue without paging--
nteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/panicking.rs:500
#115 std::panicking::try<core::result::Result<(), rustc_span::ErrorGuaranteed>, core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>>> () at library/std/src/panicking.rs:464
#116 0x0000fffff21d16ac in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/panic.rs:142
#117 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>> () at library/std/src/thread/mod.rs:528
#118 core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<rustc_interface::util::run_in_thread_pool_with_globals::{closure#0}::{closure_env#0}<rustc_interface::interface::run_compiler::{closure_env#0}<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure_env#1}>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, core::result::Result<(), rustc_span::ErrorGuaranteed>>, ()> () at library/core/src/ops/function.rs:250
#119 0x0000ffffee86a590 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#120 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#121 0x0000ffffee880238 in std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#122 0x0000ffffee5fd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
#123 0x0000ffffee665d1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

I encountered this in rust-lang/rust#113046

How to reproduce:

  • checkout https://github.com/rust-lang/rust
  • run ./x setup compiler && x test tests/ui/rustc-rust-log.rs
  • you can run rustc directly by running rustup toolchain link stage1 build/host/stage1 and then rustc +stage1
@Noratrieb Noratrieb mentioned this issue Jun 27, 2023
Merged
@davidbarsky
Copy link
Member

I don't know why this is occurring right now, but is this where rustc sets up tracing?

As an additional note, tracing-core-0.1.31 did introduce some changes that might be deadlock adjacent.

@Noratrieb
Copy link
Author

Yes, you're looking in the right place.

@hawkw hawkw added kind/bug Something isn't working crate/core Related to the `tracing-core` crate labels Jun 28, 2023
@hawkw
Copy link
Member

hawkw commented Jun 28, 2023

Well, this is definitely a bug! We'll look into it.

@nnethercote
Copy link

I just stumbled across this. Any progress?

@mladedav
Copy link
Contributor

I just went throught the stacktrace and it seems like a deadlock inside tracing-tree. Long story short, because tracing-tree uses std, it tries to log stuff while it's already logging.

It uses a lock when you try to log something, but it internally used write_fmt, which through a lot of calls ended at rustc_hir::definitions::DefPath::make, which itself tried to log something else, but the lock inside tracing_tree::HierarchicalLayer::on_evevt is already held so it deadlocks.

So I guess using the fmt layer from tracing-subscriber might be a short-term solution (although I can't guarantee there wouldn't be a similar issue) or filtering out rustc_hir::definitions (which might or might not help depending on whether there are also other tracing::event! calls that would deadlock later and it might also turn off the logs you're looking for).

Proper fix will probably have to be done in tracing-tree, but it might end up dropping recursive logs which I'm not sure how bad that would be for debugging rustc.

@mladedav mladedav mentioned this issue May 14, 2024
Merged
@mladedav
Copy link
Contributor

@nnethercote Could you try updating tracing-tree to 0.3.1 and running the tests (or whatever it was that broke you)?

This should get rid of the deadlock but note that some output logged inside the write! macros may be missing as anything called while emitting logs will have logging turned off.

@davidbarsky
Copy link
Member

I'm going to close this; this is an issue in tracing-tree.

@nnethercote
Copy link

nnethercote commented May 27, 2024 •
edited
Loading

Sorry, I should have been clearer. When I said that I "stumbled across this" I meant that I saw this line in rustc:

https://github.com/rust-lang/rust/blob/529bb2573a2a07d29329db8ff1bac8a7c6a4757f/compiler/rustc_log/Cargo.toml#L9

but I haven't actually encountered the bug in action myself.

@Nilstrieb, are you able to try tracing-tree 0.3.1 and determine if it fixes the original problem?

@klensy klensy mentioned this issue Jul 4, 2024
Open
bors added a commit to rust-lang-ci/rust that referenced this issue Jul 4, 2024
@bors
Auto merge of rust-lang#127316 - klensy:tracing-bump, r=<try>
8836382 
bump tracing

Unpins `tracing`: linked issue looks fixed, tokio-rs/tracing#2635 referencing davidbarsky/tracing-tree#82

Perf run please, as changelog says about improvements https://github.com/tokio-rs/tracing/blob/tracing-0.1.40/tracing/CHANGELOG.md https://github.com/tokio-rs/tracing/blob/tracing-core-0.1.32/tracing-core/CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crate/core Related to the `tracing-core` crate kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nnethercote @davidbarsky @hawkw @mladedav @Noratrieb