libdrgn: read kdumps (new dependecy: libkdumpfile)

[sdimitro]

Intro

Note that this is a WIP to get feedback on the design/organization of the code.
After writing this in C, I realized that it could have been easily implemented in
Python too since libkdumpfile provides wrappers for Python. I'm not sure what
would be preferable at this point but it wouldn't be a problem for me rewriting
this in Python.

Preliminary Feature Testing

Look at the jiffies of kdump in crash:

$ crash ../dump/vmlinux-4.15.0-47-generic ../dump/dump.201904221843

crash 7.2.1
Copyright (C) 2002-2017  Red Hat, Inc.
... < cropped output > ...
crash> p jiffies
jiffies = $2 = 4294918568

Using the same address to read the jiffies from drgn prototype we get the same value:

$ sudo drgn -c dump/dump.201904221843 -s dump/vmlinux-4.15.0-47-generic
could not get debugging information for:
kernel (could not find vmlinux)
kernel modules (could not find loaded kernel modules: could not find 'struct module')
drgn 0.0.1 (using Python 3.6.7)
For help, type help(drgn).
>>> import drgn
>>> from drgn import cast, container_of, execscript, NULL, Object, reinterpret
>>> from drgn.helpers.linux import *
>>> prog['jiffies']
(volatile unsigned long)4294918568
>>>

Testing that feature is indeed an optional dependency

The above testing was done by enabling the feature through setup.py like this:

--- a/setup.py
+++ b/setup.py
@@ -40,6 +40,7 @@ class my_build_ext(build_ext):
             args = [
                 os.path.relpath('libdrgn/configure', self.build_temp),
                 '--disable-static', '--with-python=' + sys.executable,
+                '--with-libkdumpfile=yes',
             ]
             try:
                 subprocess.check_call(args, cwd=self.build_temp)

Without the feature trying to open the same crash dump we get the following error:

$ sudo drgn -c dump/dump.201904221843 -s dump/vmlinux-4.15.0-47-generic
Traceback (most recent call last):
  File "/usr/local/bin/drgn", line 11, in <module>
    load_entry_point('drgn==0.0.1', 'console_scripts', 'drgn')()
  File "/usr/local/lib/python3.6/dist-packages/drgn-0.0.1-py3.6-linux-x86_64.egg/drgn/internal/cli.py", line 83, in main
    prog.set_core_dump(args.core)
ValueError: drgn configured without libkdumpfile

Work left

• finalize ifdef structure for making this an optional dependency
• figure out if automated testing is possible?
• style fixes
• copy Petr Tesarik (maintainer of libkdumpfile) in the final rounds of code review to ensure proper use of the library (and also give a heads up that we exist as an official consumer)

[sdimitro]

Seems like Travis is failing because the images do not have libkdumpfile installed

[osandov]

A few more minor comments. I think C is preferable for this so that it's supported from the C API. Thanks for the contribution!

[osandov]

As a side note, normal ELF vmcores don't include vmalloc or per CPU addresses in the program headers, so drgn currently can't read from them. It seems like libkdumpfile can, so perhaps as a followup change, we should also use libkdumpfile for normal ELF vmcores.

[sdimitro]

Asked a couple of questions above, and applied the rest of the feedback in a separate commit. As this is still a WIP, I'll leave each round of feedback as a separate commit and squash them into one commit before merging.

[osandov]

Hey, sorry for the delay. A few comments about reorganizing the conditional compilation.

[osandov]

I pushed a tweak to the configure script so that we automatically use libkdumpfile if it's available. A few final things:

• Let's rename kdump_reader.[ch] to kdump.[ch] since they have more logic than just the reader.
• I don't see where we're freeing the kdump context. We can probably just stuff it in struct program as a void * and close it if it's not NULL.
• Before I merge this, I'll reformat the code a bit to the Linux kernel coding style, but you don't need to worry about that.

[osandov]

I rebased this on top of the platform change in the devel branch. Can you please test it out? If everything works well, I'll merge it.

[sdimitro]

Just tested the PR and the output is the same as the description (I tried a few other globals and types from modules that I loaded with -s too) so from that perspective we are good to go.

I remember you mentioning on our DMs yesterday, that you went ahead and did the freeing of the kdump context yourself? I didn't see that in the changes that I pulled (neither the renaming s/kdump_reader/kdump/ for the new files). Do you want me to go ahead and do it. or you do have changes but they are still local?

[osandov]

Ah, I pushed the changes to the devel branch on my repo rather than updating your branch.

[sdimitro]

Gave the devel branch a run. Works the same :-)

[osandov]

Merged, thanks!