dtrace's ustack() function cannot unwind stack frames on FreeBSD since 1.56.0

[asomers]

One of the best ways to profile Rust code is with dtrace, as described at https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#DTrace . As recently as Rust 1.55.0 that worked fine. But with Rust 1.56.0 the stack just isn't there. Dtrace can only see a single Rust-level stack frame. For example:

Code

With Rust 1.55.0

$ sudo dtrace -x ustackframes=100 -n 'profile-199 /pid == $target && arg1/ {@[ustack()] = count();}' -c "target/debug/deps/test-bc98e078c2bbb61b"
    ....
              libc.so.7`__je_malloc_mutex_prefork+0x124
              libc.so.7`__je_arena_prefork7+0x73
              libc.so.7`_malloc_prefork+0x15b
              libthr.so.3`0x392e4a8c4686
              libthr.so.3`_fork+0x18
              test-dad15ed382b075cf`nix::unistd::fork::h358225d652a86eab+0xe
              test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::hb93c7cdf2b79d680+0x36
              test-dad15ed382b075cf`test::test_unistd::test_fork_and_waitpid::_$u7b$$u7b$closure$u7d$$u7d$::h329a121974ff9291+0x11
              test-dad15ed382b075cf`core::ops::function::FnOnce::call_once::h2261827bcba63036+0x11
              test-dad15ed382b075cf`test::__rust_begin_short_backtrace::hefb7644d11da2ff9+0xa
              test-dad15ed382b075cf`test::run_test::run_test_inner::_$u7b$$u7b$closure$u7d$$u7d$::hdaa0fb71aac8d97e+0x2f3
              test-dad15ed382b075cf`std::sys_common::backtrace::__rust_begin_short_backtrace::h8bcc057a546c1087+0xce
              test-dad15ed382b075cf`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf7d978d08be459d0+0x6a
              test-dad15ed382b075cf`std::sys::unix::thread::Thread::new::thread_start::h6b52ca0eca213387+0x2b
              libthr.so.3`0x392e4a8c3a7a
    ...

With Rust 1.56.0

$ sudo dtrace -x ustackframes=100 -n 'profile-199 /pid == $target && arg1/ {@[ustack()] = count();}' -c "target/debug/deps/test-bc98e078c2bbb61b"
    ...
              libc.so.7`__je_malloc_mutex_prefork+0x124
              libc.so.7`__je_arena_prefork7+0x73
              libc.so.7`_malloc_prefork+0x15b
              libthr.so.3`0x1106cebc6686
              libthr.so.3`_fork+0x18
              test-b377ad62cc9e0624`nix::unistd::fork::hbf1ed55b658aa870+0xa
              0x8
              0xcccccccccccccccc
    ...

Version it worked on

It most recently worked on: Rust 1.55.0

Version with regression

rustc 1.56.0 (09c42c458 2021-10-18)

Workaround

The old behavior can be restored by compiling with RUSTFLAGS="-C force-frame-pointers". This flag was added by #48785 , but that was a long time ago. There is nothing in the 1.56.0 release notes, nor any recent commit messages, that mention anything about frame pointers. Was this change therefore accidental? Or could it have been a by product of switching to LLVM 13?

[nagisa]

Does it still fail if you set -Cforce-unwind-tables=y (but not frame pointers)?

[nagisa]

triage note: this is most likely unrelated to debuginfo. Usual mechanisms for unwinding are uwtables or frame pointers, both mechanisms are independent of debuginfo.

@rustbot label -A-debuginfo

[asomers]

Does it still fail if you set -Cforce-unwind-tables=y (but not frame pointers)?

Yes, it still fails if I use RUSTFLAGS="-Cforce-unwind-tables=y"

[nagisa]

And if you br/bt in a similar location in gdb or use other unwinding implementations?

[asomers]

Yes, gdb can print stack traces even without a custom RUSTFLAGS.

[nagisa]

So the short summary is basically: something changed in how we emit the uwtables between 1.55 and 1.56. uwtables are clearly still “valid”, at least for the purposes of the unwinder used by gdb, but not so for the purposes of whatever unwinding implementation dtrace uses. Ultimately it is LLVM that's emitting the tables so…

• I wonder if dtrace fails the same way with clang 13 (note: make sure frame pointers are disabled and uwtables are enabled)?
  • If reproducible, send reproducer to LLVM directly and/or report it to maintainers of llvm/clang packages in freebsd;
• I assume the target here is a x86_64. You should be able to find the uwtables in a dedicated section (.eh_frame), it would be worthwhile to compare the shape of the uwtables.
  • readelf --debug-dump=frames can help here
  • having a minimal example is also useful to ensure you aren't bogged down by irrelevant changes.

[apiraino]

Assigning priority as discussed in the Zulip thread of the Prioritization Working Group.

@rustbot label -I-prioritize +P-medium