Add support for debuginfod lookup/download of separate debuginfo

[fche]

Around four years ago, libbacktrace learned to handle external (split) debuginfo, whereby it searches various distro standard locations by build-id. If this search is unsuccessful, it would be good to try using the increasingly available debuginfod facility to attempt downloading of a corresponding dwarf file from servers.

For licensing or other reasons, you may not be interested in using the C elfutils debuginfod client library, but you may find spawning the command line tool debuginfod-find suitable. https://www.mankier.com/1/debuginfod-find

See also: https://sourceware.org/elfutils/Debuginfod.html .

[ianlancetaylor]

Hi Frank. The main issue is that libbacktrace is designed to run in a signal handler, which means it can't allocate memory using any standard mechanism like malloc. Incorporating an HTTPS client into libbacktrace is pretty much a non-starter. But it might be OK to call execve to run debuginfod-find.

[ianlancetaylor]

Although one issue is that I wouldn't want people to have unexpected waits before a backtrace is printed. That might be tolerable if the backtrace gets symbolic information, but a wait to contact a network server in order before getting limited information would be annoying. This might have to be an opt-in facility, or at least an opt-out one.

[fche]

Good points all around. Other than the env-var based opt-in ($DEBUGINFOD_URLS), libbacktrace or the user could set extra variables to limit time/space of downloads ($DEBUGINFOD_MAXTIME in seconds / _MAXSIZE in bytes).

[ianlancetaylor]

For clarification, are DEBUGINFOD_MAXTIME and DEBUGINFOD_MAXSIZE existing environment variables supported by the debuginfod-find program? Thanks.

[fche]

They are in pre-release elfutils (so will be 0.186 onward, coming by November or so), and will be understood by all debuginfod clients, including the command line front-end.

[fche]

(elfutils 0.186 was released a few weeks ago)

[wolfpld]

Adding support for debuginfod to libbacktrace can be performed in the following way.

First, you extend the elf_open_debugfile_by_buildid function, which is already looking for external debug symbols, in the form usually provided by the debuginfo packages. If these are not found, we may relegate the the task of getting the descriptor to libdebuginfod.

wolfpld/tracy@4559120

Here the GetDebugInfoDescriptor function basically calls debuginfod_find_debuginfo. Do note that at this point in program flow, a valid descriptor must be returned, as you won't get another chance to do so. libbacktrace is caching the information for each executable image, so if you return -1 (i.e. a failure), the queried image won't be checked again. This random bit of trivia is essential to know, if we consider that elf_open_debugfile_by_buildid may be called from a callback of dl_iterate_phdr, and libdebuginfod may want to use libcurl to download the debug information. These two don't mix, as you will surely discover.

This problem can be worked around by performing a dry run of dl_iterate_phdr, in which we just save the required information in some kind of data structure. Then, we may run the originally intended loop, but without the constraints of dl_iterate_phdr. libbacktrace requires only two data fields provided by dl_iterate_phdr.

wolfpld/tracy@7e8961d

With these changes libbacktrace is able to provide debuginfod information.

[ianlancetaylor]

Thanks, but I have a feeling that those approaches will fail badly when libbacktrace is called from a signal handler.

[wolfpld]

Not every application of libbacktrace involves signal handlers.

[ianlancetaylor]

That is true. But if we add calls out to debuginfod, then those uses of libbacktrace that do use signal handlers seem likely to break.

[wolfpld]

Yes. A callback system might be used instead, like in the example I have posted. User of the library might then just return -1, if proper behavior in signal handling is expected (this also conveniently leaves it up to the user to decide if this should be the case). This has the added benefit of offloading the initialization and cleanup of libdebuginfo to the user.