(Ubuntu's) lldb doesn't print stack variables

[japaric]

Note: This issue isn't Ubuntu specific as I observe the same issue on the obscure Linux distribution I use everyday.

STR

$ cat foo.rs
fn main() {
    let x = 1;
    let y = frob(x);
}

#[no_mangle]
fn frob(x: i32) -> u32 {
    let y = x as u32;
    y
}

$ rustc -g foo.rs

$ lldb ./foo
(lldb) target create "./foo"
Current executable set to './foo' (x86_64).
(lldb) b frob
Breakpoint 1: where = foo`frob + 7, address = 0x0000000000005247
(lldb) r
Process 4499 launched: './foo' (x86_64)
Process 4499 stopped
* thread #1: tid = 4499, 0x0000555555559247 foo`frob + 7 at foo.rs:8, name = 'foo', stop reason = breakpoint 1.1
    frame #0: 0x0000555555559247 foo`frob + 7 at foo.rs:8
   5
   6    #[no_mangle]
   7    fn frob(x: i32) -> u32 {
-> 8        let y = x as u32;
   9        y
   10   }
(lldb) p &x
(lldb) p &y
(lldb) p x
(lldb) frame variable
(lldb)

lldb doesn't print stack variables, but gdb does:

$ gdb ./foo
(...)
(gdb) b frob
Breakpoint 1 at 0x5247: file foo.rs, line 8.
(gdb) r
Starting program: /home/japaric/foo
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, foo::frob (x=1) at foo.rs:8
8           let y = x as u32;
(gdb) p &x
$1 = (i32 *) 0x7fffffffe3fc
(gdb) p &y
$2 = (u32 *) 0x7fffffffe3f8
(gdb) p x
$3 = 1
(gdb) info args
x = 1
(gdb) info locals
y = 4286582784

And lldb can print stack variables just fine when used with C programs:

$ cat foo.c
int frob(int);

int main() {
  int x = 1;
  int y = frob(x);

  return y;
}

int frob(int x) {
  unsigned int y = x;
  return y;
}

$ gcc -g foo.c
$ lldb ./a.out
(lldb) target create "./a.out"
Current executable set to './a.out' (x86_64).
(lldb) b frob
Breakpoint 1: where = a.out`frob + 7 at foo.c:11, address = 0x000000000040051e
(lldb) r
Process 4605 launched: './a.out' (x86_64)
Process 4605 stopped
* thread #1: tid = 4605, 0x000000000040051e a.out`frob(x=1) + 7 at foo.c:11, name = 'a.out', stop reason = breakpoint 1.1
    frame #0: 0x000000000040051e a.out`frob(x=1) + 7 at foo.c:11
   8    }
   9
   10   int frob(int x) {
-> 11     unsigned int y = x;
   12     return y;
   13   }
(lldb) p &x
(int *) $0 = 0x00007fffffffe63c
(lldb) p &y
(unsigned int *) $1 = 0x00007fffffffe64c
(lldb) p x
(int) $2 = 1
(lldb) frame variable
(int) x = 1
(unsigned int) y = 0

I'm not sure what's going on. Do I need some extra steps to make lldb fully work with Rust programs? I tried rust-lldb but I get the same behavior.

Meta

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.10
DISTRIB_CODENAME=wily
DISTRIB_DESCRIPTION="Ubuntu 15.10"

$ lldb -v
lldb version 3.8.1 ( revision )

$ gdb -v
GNU gdb (Ubuntu 7.10-1ubuntu2) 7.10
(...)

$ rustc -Vv
rustc 1.10.0-nightly (2174bd97c 2016-04-14)
binary: rustc
commit-hash: 2174bd97c1458d89a87eb2b614135d7ad68d6f18
commit-date: 2016-04-14
host: x86_64-unknown-linux-gnu
release: 1.10.0-nightly

cc @michaelwoerister

[michaelwoerister]

Yeah, I noticed this too and newer versions of LLDB seem to have the same problem on OSX. So far, LLDB hasn't gotten any attention outside of OSX but maybe this problem is the same underlying problem. I'll start debugging this next week when I find some spare time.

[jasonmolenda]

it's probably something about the debug information (DWARF) for the function that's causing the problem, unlikely to be platform (Ubutu/linux/etc) specific. When frame variable reports nothing, it's clear lldb doesn't know of any variables there. Usually you'd start with image lookup -va $pc to see exactly where lldb thinks it is, then run readelf on your binary and look at the DWARF there. You may want to ask for help on the lldb-dev mailing list over at http://lldb.llvm.org reading the DWARF, it can be nonobvious if you're familiar with the encoding.

[jasonmolenda]

I'd recommend checking out a copy of the current lldb sources from http://lldb.llvm.org and seeing if that works better. The linux support in lldb has matured greatly over the last year and depending on when your linux distro's copy of lldb was made, it may be missing important support for e.g. "dwo" files.

[michaelwoerister]

So, I spent the whole day trying to track this issue down and finally found it in the last place I would have suspected it: LLDB seems to have changed its behavior when encountering a compilation with an unknown DW_AT_language attribute. Previously, it did something similar to GDB, i.e. trying to evaluate expression as if they where C/C++ -- now it doesn't do anything :/

This is a bit of a problem since I'd prefer not to emit DW_LANG_C_plus_plus as the language for Rust binaries, but now there are LLDB versions out in the wild that just won't work otherwise. We'll need to come up with a strategy to deal with this situation.

cc @rust-lang/compiler @rust-lang/tools

[alexcrichton]

Whoa, nice find! Could you expand on the downsides of just marking ourselves as C++ for now? It seems like that may be our only course of action in the near future, unless there's perhaps a way for rust-lldb to "do the right thing" like set it up as a C++ binary even through it says it's a Rust binary?

Looking forward, maybe we could try to beef up rust-lldb to do version detection and "do the right thing"?

[jasonmolenda]

Nice detective work. The proper solution would be to add a rust parser to lldb (see source/Plugins/ExpressionParser/Go/ in the lldb repository for an example of a language that does not use the clang front-end) - a less satisfying approach is to tell lldb to try parsing Rust as if it were C++ e.g. see the RenderScript support in ClangASTContextSupportsLanguage() in source/Symbol/ClangASTContext.cpp. I don't know rust so I don't know how well it maps on to the C/C++/Objective-C syntax - it seems like something that would be fragile and fail in confusing ways to users.

You may want to ask about this "if we don't know what language it is, assume a C family language" assumption on the lldb-dev mailing list. I wasn't paying attention when that change went in, but I imagine it was made because the chance that the debugger would behave poorly was high.

Another example of a language added to lldb, but not in the lldb.llvm.org repository, is the swift support. All of that support is over at swift.org (github repositories) - the swift language support pretty cleanly drops on top of lldb (the model being designed os that additional languages can be added by other groups). Like the C-based languages that use clang as the front end, the swift support uses the swift compiler as its front end. This is a large amount of code, of course, and assumes there's a compiler structured in the form of a library and probably having to emit LLVM IR etc.

[michaelwoerister]

@jasonmolenda Thanks for the pointers to other language implementations.

The proper solution would be to add a rust parser to lldb.

That's definitely the long term solution to this problem.

I wasn't paying attention when that change went in, but I imagine it was made because the chance that the debugger would behave poorly was high.

I assume it just fell out of adding support for language plugins.

[michaelwoerister]

Could you expand on the downsides of just marking ourselves as C++ for now?

The practical downsides would be negligible, I guess. We couldn't rely on debuggers doing anything Rust specific before and C++ is the closest thing. I thought, there might be some benefit in choosing a value that is definitely not a known language (e.g. DW_AT_language=0x0), so that debuggers might try to activate some more fallback mechanisms when trying to deal with Rust. But at the moment I can't think of anything specific where this might be the case.

In the long run we just need a proper language plugin for LLDB.

there's perhaps a way for rust-lldb to "do the right thing" like set it up as a C++ binary even through it says it's a Rust binary?

I don't think there's a way to do that, unfortunately.

[januszn]

there's perhaps a way for rust-lldb to "do the right thing" like set it up as a C++ binary even through it says it's a Rust binary?

You can do settings set target.language c++ but the TypeSystem abstraction introduced in LLDB 3.8 is pervasive in that overriding the language on the "target" level is not enough - you also need to do it for reading the symbols and I don't think there's a way exposed to do this - I've done that in a debugger (and it works then).

PS. Apologies for butting in, it's just that I'm interested in this issue as well and I've been debugging it today out of curiosity (see https://users.rust-lang.org/t/rust-lldb-not-printing-out-local-variables/3980/6 and http://stackoverflow.com/questions/36621130/how-can-i-inspect-variable-values-while-debugging-msvc-abi-rust-programs).

[Manishearth]

Could we ship a language plugin with rust that just redirects lldb to C++ (and later improve on it so it's smarter)?

[jasonmolenda]

My two cents: The right solution is for the rust files' debug info to use DW_LANG_Rust (0x1c) as their language code. gdb and lldb can be modified to handle DW_LANG_Rust as if it were C++ if that's the closest approximation. I mentioned in an earlier comment that the RenderScript language support in lldb behaves this way, see ClangASTContextSupportsLanguage() in source/Symbol/ClangASTContext.cpp. Eventually when someone wants to add more full-fledged support for Rust expression parsing, they can enhance the debuggers to do that.

Emitting no DW_AT_language attribute, or putting DW_LANG_C_plus_plus_11 in there is IMO a poor short-term hack. You may want to do that temporarily because of existing, deployed debuggers that won't handle DW_AT_language + DW_LANG_Rust right now - but I'd be getting patches in to all the relevant tools (gdb, lldb, others?) to do the right thing when they do eventually see a DW_LANG_Rust.

[Manishearth]

@jasonmolenda "All of that support is over at swift.org (github repositories) - the swift language support pretty cleanly drops on top of lldb (the model being designed os that additional languages can be added by other groups). "

So does swift ship a dylib plugin or something? Can we do the same thing in a bare-bones way and just have it redirect to C++ for now?

[Manishearth]

If that's not possible, I suggest we patch lldb asap so that it treats DW_LANG_RUST as C++, and by the time that reaches a release we can switch to emitting DW_LANG_RUST by default so that the gdb thing is more widely useful. If this language switch flag is to be kept around for a long time we'd need 'cargo integration and all and I'd really prefer to get rid of it as fast as possible.

[jasonmolenda]

lldb is hosted over at http://lldb.llvm.org (not github, unfortunately). Check out the sources (svn or there's a read-only git mirror) and make the change I suggest above -- verify that it works correctly & is sufficient for your rust program when it calls itself DW_LANG_Rust in the debug information. Normally you'd then you'd go through the patch review system there - add a Differential Review Code at http://reviews.llvm.org and Cc lldb-commits on the patch. It's a bit of an involved process for a one-time commit. If you'd prefer, once you come up with a patch that you've tested as working correctly for your sample rust programs, send it to the lldb-commits mailing list and I'll help move it along.

[Manishearth]

Thanks. I personally don't have enough space right now for an extra LLVM+LLDB build (no time either, and already working on gdb stuff).

Is anyone else up for this?

[michaelwoerister]

@Manishearth I'll look into it.
@jasonmolenda Thanks for offering your help, it's very much appreciated!

[michaelwoerister]

To give an update on this: The issue with LLDB not printing anything is mitigated in the open-source version of LLDB for a few weeks now -- i.e. the "old" behavior of falling back to Clang-mode for Rust has been restored and will stay in place until a proper Rust plugin gets written for LLDB.

The versions of LLDB distributed with XCode over the last few months, on the other hand, do not include this fix. I don't have any insight into how fast the fix will show up in Apple's binaries.

[jasonmolenda]

I cherry-picked the change on to the swift 3.0 branch for lldb, and it is included in the Xcode 8.0 preview release to apple developers that went public today.

[vadimcn]

Ah, nice! Thanks for the update!

But what about Linux distros that bundle LLVM 3.8? (is that when the breakage started?)

[jasonmolenda]

Ah, I'm afraid I don't follow the llvm release branches, mostly just the apple side of things, so I can't comment on that. I knew it was important to get this patch into distributions and made sure it would be in Xcode 8, but didn't think to look into the llvm releases.

[vadimcn]

This works in LLDB 3.9+

[maccoda]

LLDB 3.9+ is able to be installed for Ubuntu using the setup described here, or using the answers posted here