Make using separate debugging info easier.

[kevincox]

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

The previous iteration of debug info required that all of the packages
were "installed". This is suboptimal because "installing" all of the
programs you want to debug as well as the libraries that they use.

This method enables debug symbols for all loaded files as long as those
symbols are present in the nix store, as well as a command to download
those symbols from binary caches for all loaded object files.

This is implemented by adding a new section to the executable witch
contains the path to the debug symbols. An extension that is auto-loaded
by GDB then uses this information to automatically load these symbols
whenever the object file is loaded.

Future work:

• Find a way to make all packages build with debug symbols by default.
• Find a better way to "attach" debug symbols to packages (weak references?)

[mention-bot]

By analyzing the blame information on this pull request, we identified @edolstra, @peti and @vcunat to be potential reviewers

[bjornfor]

Interesting! I would very much like to have this feature.

[edolstra]

Doesn't including the path to the debug symbols in the executables make the executables depend on the debug symbols? That would defeat the purpose of separate debug info...

[bjornfor]

@edolstra: If so, maybe we can add support for "weak store references" in nix? Or some way to simply ignore certain store path references (to prevent them being hard dependencies)?

[kevincox]

@edolstra Yes. This is currently worked around by adding a space in the middle of the hash portion of the store path. This has downsides because now the debug packages don't automatically get pushed to a cache with nix-push. This might be a legitimate use case for weak references as @bjornfor mentions.

On a vaguely similar note. Because the debug info is compressed any references it holds are not recognized. This isn't a big deal right now because it appears that the only thing that gets referenced is the package it came from but if we discover a solution for embedding proper source paths it will become a nuisance. Is there any support for handling references that get mangled in some way?

[kevincox]

it appears that the only thing that gets referenced is the package it came from

Upon closer inspection it also references all of the libraries the program is linked with. This probably still isn't a big deal as the package itself should reference these as well.

[edolstra]

@kevincox Ah, thanks for the clarification. However, hiding references in this way is essentially a weak reference, which is problematic because it may make derivations produce a different result depending on whether a -debug store path is present or not. This is not a theoretical concern: we use Nix builds at work to run automated tests, where we invoke gdb to produce stack traces on failure. So unless chroot builds are enabled, this could produce different output if debug symbols are present in the Nix store of the build machine.

[kevincox]

@edolstra I see what you are saying. You would get different stack traces if the symbols were present in the store. While there are downsides to this I think that this could probably be considered a benefit at the same time. We don't want to stop you from debugging your code, but if available you will get a better experience. This is where legitimate weak reference scheme would add benifit.

Also I don't know how you are generating stack traces but you can either download the debug info before grabbing the trace (and now your traces are more complete) or if you are taking core dumps it doesn't matter if there is the info present on the target machine, because you can download the debug symbols to the machine you are debugging on (assuming they are available on a binary cache)

Also this isn't much worse then the existing method as your stack traces would change based on what is "installed" at the time, rather then just the store paths themselves.

So while I see the downside of determinism I think the benefit here is clear and we should look into ways to get the best trade-offs.

[kevincox]

I remove the point about source inclusion because it is a huge file size and it isn't incredibly useful because it is fairly easy to acquire the source yourself. If one day we want to add it it can be done later.

What are the thoughts about inclusion? I've been using this on my machines and it is very helpful for debugging crashes.

[vcunat]

@kevincox: I don't see into implementation details of this PR, but what about adding references from the debugging outputs instead of the regular ones? That wouldn't require any reference hiding or weak references, and I believe it's reasonable that the debug outputs would (transitively) reference debug outputs of the whole library-closure.

[kevincox]

what about adding references from the debugging outputs instead of the regular ones

I don't follow what you are saying. We need a reference from the regular output to the debug output so that the debugging symbols can be found.

[vcunat]

I meant that you would need to "install" (in some way) the app.debug output but nothing else (not the depending libraries).

[kevincox]

This isn't really feasible as the links in the debugging information point to the linked binaries, not their debugging information. Also for me the single command install without looking up paths is the killer feature for me. I don't want to have to remember when, where and with what flags I compiled the program. Basically in my flow I never remember the path of the debug info, instead it is automatically pulled from the binary at debug time.

[vcunat]

I think it would still be automatizable, as e.g. for stuff built without substitutes we've got:

nix-store --realize "$(nix-store -q --deriver /path/to/your/executable)!debug"

but certainly I don't think there's any "perfectly pure" way as simple as this PR (which makes it tempting).

[kevincox]

@vcunat IIUC that would require that you have the derivation spec on the machine that runs the command. Maybe I'm using Nix "unusually" but I build all of my packages on a CI server then upload to a binary cache, only storing the store path of the "useful" output(s). This means that the derivation is gone forever (save checking out the nixpkgs version and git version of my package and building it again).

The way I have it now will work as long as you have any way to build the debug output, whether that is the derivation, or access to any binary cache with the debug output.

[mmahut]

Are there any updates on this pull request, please?

[edolstra]

I'll close this because relying on hidden references is not a good idea IMHO (for instance it will break with hash rewriting). You can get debug symbols automatically by using dwarffs.