Support setting custom vmcoreinfo

[brenns10]

This branch has sat on the back burner for a while, but new discussion in #350 led me to update it and get it ready to merge. It adds the ability to give drgn some data to be interpreted as the vmcoreinfo when the Program is created. Later, when a core dump is attached, drgn will respect the pre-existing info rather than trying to get it from the core dump.

There's a bit of a delicate dance to be had with libkdumpfile, because it also wants to use the vmcoreinfo to find the page tables. For vmcores missing the vmcoreinfo, it seems that libkdumpfile doesn't know how to find their architecture, so we need to manually provide the architecture, then the dump FD, then the vmcoreinfo.

While I was at it, I threw in RISC-V support for libkdumpfile, which has been available since 0.5.4, but since we have some if-statements that enumerate all the libkdumpfile architectures, we never updated those and thus haven't been able to use it (I think... I don't actually have a RISC-V core dump handy).

---

Testing this is of course the important part. One obvious candidate where this code makes drgn work is hypervisor core dumps that are missing vmcoreinfo. QEMU will produce them if you don't have -device vmcoreinfo, which drgn already notes and produces a nice error message about.

Thankfully, drgn's own test framework runs QEMU without that option, so it's easy to generate a good test vmcore:

python -m vmtest.vm -k 6.9*
...
# Issue key sequence: Ctrl-A C to switch to the QEMU monitor
(qemu) dump-guest-memory file.elf
## OR
(qemu) dump-guest-memory -z file.kdump.flat

Running drgn against it will fail:

$ drgn -c file.kdump.flat -s build/vmtest/x86_64/kernel-6.9.1-vmtest29.1default/build/vmlinux
drgn 0.0.26 (using Python 3.9.18, elfutils 0.190, with libkdumpfile)
warning: the given file is in the makedumpfile flattened format; if open fails or is too slow, reassemble it with 'makedumpfile -R newfile <oldfile'
Traceback (most recent call last):
  File "/home/stepbren/.local/bin/drgn", line 33, in <module>
    sys.exit(load_entry_point('drgn', 'console_scripts', 'drgn')())
  File "/usr/lib64/python3.9/site-packages/drgn/cli.py", line 270, in _main
    prog.set_core_dump(args.core)
Exception: kdump_vmcoreinfo_raw: linux.vmcoreinfo.raw is not set

$ drgn -c foo.elf -s build/vmtest/x86_64/kernel-6.9.1-vmtest29.1default/build/vmlinux
drgn 0.0.26 (using Python 3.9.18, elfutils 0.190, with libkdumpfile)
error: unrecognized QEMU memory dump; for Linux guests, run QEMU with '-device vmcoreinfo', compile the kernel with CONFIG_FW_CFG_SYSFS and CONFIG_KEXEC, and load the qemu_fw_cfg kernel module before dumping the guest memory (requires Linux >= 4.17 and QEMU >= 2.11)

But with this branch, you can use my dumpphys tool like below to get a text file containing the vmcoreinfo:

dumpphys -c file.kdump.flat -o file.vmcoreinfo -i
## or
dumpphys -c file.elf -o file.vmcoreinfo -i

And then, you can go ahead and run drgn, but add new arguments --vmcoreinfo file.vmcoreinfo to specify the vmcoreinfo, and --architecture X86_64 to specify the architecture (since libkdumpfile cannot detect it). At that point, drgn works like normal.

---

For ELF vmcores, drgn requires some added bootstrapping/knowledge to find the physical address of the root page table (i.e. swapper_pg_dir). I've implemented this ability within the x86_64 page table iterator. Other architectures will need this support added. libkdumpfile already seems to have this ability for x86_64, so for other architectures it stands to reason that DRGN_USE_LIBKDUMPFILE_FOR_ELF=1 might help them avoid the following error:

>>> prog["slab_caches"]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/stepbren/repos/drgn/drgn/cli.py", line 141, in _displayhook
    text = value.format_(columns=shutil.get_terminal_size((0, 0)).columns)
_drgn.FaultError: recursive address translation; page table may be missing from core dump: 0xffffffff9662aff8

[iostapyshyn]

The current known issue with this branch is that this won't work with ELF vmcores that drgn opens natively. Drgn expects to have virtual address mappings provided for it, at least for the linear/direct map. When they're not there, drgn will fail with a recursive page translation error:

The same issue exists with guest memory dumps from QEMU that don't include VM translations. Currently, drgn (without libkdumpfile) works out of the box with QEMU dumps generated with the paging flag (dump-guest-memory -p). With the flag, QEMU performs page table walks and replicates the VM mapping via ELF segments in the dump. This appears to be slow and highly inefficient: In the best case this takes about a minute and the resulting ELF has just a little over 60000 segments, but in some cases the QEMU process hangs indefinitely (I suppose while the guest is executing a process with a huge address space).

The physical memory dumps (without paging) are much faster and work with drgn when DRGN_USE_LIBKDUMPFILE_FOR_ELF=1.

Without libkdumpfile, read_memory_via_pgtable starts from swapper_pg_dir, which is stored in VMCOREINFO as a virtual address. If the core dump doesn't have a translation for swapper_pg_dir, the page table walk will attempt to translate that address again via read_memory_via_pgtable, resulting in recursive address translation fault.

I propose applying the kernel mapping offset to swapper_pg_dir in such cases, which I've implemented in this commit as a PoC for x86_64, but haven't checked other architectures yet. I'll gladly work on a general solution though.

[brenns10]

QEMU dumps generated with the paging flag (dump-guest-memory -p)

Ooof, yeah, this option is heavily discouraged due to performance. Also, a malicious guest could cause a bit of a denial of service by finagling the page tables in just the right way so that the page mappings are way too large & complex for QEMU to handle.

But, drgn should also work out-of-the-box with vmcores generated by dump-guest-memory without -p, so long as the host QEMU is started with -device vmcoreinfo and the guest Linux version is > 4.17 configured with CONFIG_FW_CFG_SYSFS and CONFIG_KEXEC (which are pretty universally enabled). See this message. If that's not the case, that's definitely a new bug to investigate. ((Note that DRGN_USE_LIBKDUMPFILE_FOR_ELF=1 may still be required for such ELF vmcores.))

I propose applying the kernel mapping offset to swapper_pg_dir in such cases, which I've implemented in this commit as a PoC for x86_64, but haven't checked other architectures yet. I'll gladly work on a general solution though.

Oops, I hope you didn't spend too long on that since I had a similar (but less polished) workaround in my vmcoreinfo_wip branch. Glad to know we came a similar solution :D

That said, I like your idea of moving the fixes into the page table iterator, that can allow the architecture code to be self-contained.

-			table_physical = false;
+			table_physical = true;

The above is from your changes to the x86_64 pgtable iterator, I think there's a risk that unconditionally setting table_physical = true here may result in incorrect behavior when reading userspace pages, since in that case the page table comes from the pgd field of struct mm which is always a virtual address. You could probably adjust this to only happen when pgtable == swapper_pg_dir, something like this:

diff --git a/libdrgn/arch_x86_64.c b/libdrgn/arch_x86_64.c
index a0a51c36..d7a2a4a4 100644
--- a/libdrgn/arch_x86_64.c
+++ b/libdrgn/arch_x86_64.c
@@ -616,7 +616,10 @@ linux_kernel_pgtable_iterator_next_x86_64(struct drgn_program *prog,
 	for (;; level--) {
 		uint64_t table;
 		bool table_physical;
-		if (level == levels) {
+		if (level == levels && it->it.pgtable == prog->vmcoreinfo.swapper_pg_dir) {
+			table = it->it.pgtable + prog->vmcoreinfo.phys_base - START_KERNEL_MAP;
+			table_physical = true;
+		} else if (level == levels) {
 			table = it->it.pgtable;
 			table_physical = false;
 		} else {

That way the changes are entirely contained within the arch-specific section of the page table code, and we can add similar code for other architectures as we see fit. I've pushed a new commit with this change and added a coauthor tag for you on it.

[osandov]

Thanks for this! I kind of hate that this is necessary, but that's not your fault, and I'm happy with how you handled it. A few minor comments throughout. The last commit also needs a commit message.

[osandov]

From my understanding of the libkdumpfile documentation, blob owns vmcoreinfo now, so this would cause kdump_block_decref() to double free, right?

Suggested change

free(vmcoreinfo);

[brenns10]

Yeah, you're definitely correct, and it really obviously looks wrong too. Good catch.

More interestingly however is that the libkdumpfile code seems to indicate that even in the error case, the reference to the blob is already stolen, and thus we cannot free it in our error path. See this comment. However, there are clearly cases of errors inside of kdump_set_attr() where the function returns and the blob is not decref'd. In summary: on some failures, we can't even decref the blob. On other failures, we should. And there's no way to tell 😜

So my vote here (I know it's gross) is to heavily comment this and get rid of both the free(vmcoreinfo) and the kdump_blob_decref().

[brenns10]

Ok, I filed ptesarik/libkdumpfile#81 for Petr, we'll see if he agrees too. Assuming he does, I'll remove the error handling and document the leak .

[osandov]

If kdump_set_attr() is supposed to take ownership, then yes, we should definitely remove kdump_blob_decref(), and I don't think we even need a comment; we just need to report any leaks as a libkdumpfile bug.

[brenns10]

Ok, dropped the comment and the kdump_blob_decref(), we can revisit it if Petr says the ownership semantic is supposed to be different.

[osandov]

A couple more things I noticed in this round.

[osandov]

Merged, thanks!