Use in-vm kernel for Whonix (and possibly Kicksecure)

[hoppity2]

How to file a helpful issue

Qubes OS release

Qubes 4.2

Brief summary

The added kernel boot parameters from Kicksecure's/Whonix's security-misc don't get added to the kernel upon VM boot.

Steps to reproduce

These can be reproduced in the templates as well as other VMs in general, but I'll just use template VMs here for simplicity.

1. Startup a Whonix GW, Whonix WS and Debian (standard) template. (Assumption all have kernels provided by Dom0.)
2. Notice the kernel boot parameters or lack of boot parameters in /etc/default/grub.d, in particular 40_cpu_mitigations.cfg and 40_kernel_hardening.cfg. ls /etc/default/grub.d
3. Open sudo journalctl -r search for the start of the boot in the logs (search for: Command line:).
4. Notice the lack of extra kernel boot parameters.
5. Clone the Debian standard template.
6. In the cloned Debian standard template install security-misc from Kicksecure/Whonix.
7. Reboot this template, again still with the Dom0 provided kernel notice again how the parameters listed above are missing.
8. In the cloned Debian standard template settings set it to use the template provided kernel (provided by qube). Apply this. You may need to change the HVM virtualization to get it to boot.
9. Restart the cloned Debian standard template with security misc in.
10. Look for the kernel parameters in journalctl for this boot, notice that the extra kernel params in /etc/default/grub.d are present.

For example:

• Whonix GW:
  • Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 clocksource=tsc xen_scrub_pages=0 swiotlb=2048 apparmor=1 security=apparmor
• Whonix WS:
  • Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 clocksource=tsc xen_scrub_pages=0 swiotlb=2048 apparmor=1 security=apparmor
• Debian (standard), before security-misc install:
  • Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 clocksource=tsc xen_scrub_pages=0 swiotlb=2048 apparmor=1 security=apparmor
• Debian (standard), after security-misc install:
  • Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 clocksource=tsc xen_scrub_pages=0 swiotlb=2048 apparmor=1 security=apparmor
• Debian (standard), after security-misc install, after changed to internal template kernel:
  • Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-27-amd64 root=/dev/mapper/dmroot ro xen_scrub_pages=0 root=/dev/mapper/dmroot console=tty0 console=hvc0 swiotlb=8192 noresume clocksource=tsc mitigations=auto,nosmt nosmt=force spectre_v2=on spectre_bhi=on spec_store_bypass_disable=on l1tf=full,force mds=full,nosmt tsx=off tsx_async_abort=full,nosmt kvm.nx_huge_pages=force l1d_flush=on mmio_stale_data=full,nosmt retbleed=auto,nosmt gather_data_sampling=force reg_file_data_sampling=on slab_nomerge slab_debug=FZ init_on_alloc=1 init_on_free=1 page_alloc.shuffle=1 pti=on randomize_kstack_offset=on vsyscall=none debugfs=off kfence.sample_interval=100 vdso32=0 amd_iommu=force_isolation intel_iommu=on iommu=force iommu.passthrough=0 iommu.strict=1 efi=disable_early_pci_dma random.trust_bootloader=off random.trust_cpu=off extra_latent_entropy loglevel=0 quiet

Expected behavior

If I install security-misc or if it's already installed the kernel boot parameters are applied irrespective if you are using a kernel in the template or one provided by Dom0.

Actual behavior

When using the Dom0 provided kernel in a VM it doesn't apply the security-misc's kernel parameters located in /etc/default/grub.d.

A workaround for this is to simply copy the boot params into Dom0 and set them with qvm-prefs [VM NAME] kernelopts "[KERNEL PARAMS]". However unfortunately kernelopts can only hold up to a maximum of 512 characters, so there isn't enough characters to fit them all in, for future proofing in the meantime perhaps increase this up to say 5120 characters.

Other

Searched on Qubes issues and Kicksecure security-misc issues, couldn't find a similar issue.

---

Issue title adjusted based on the decision for the solution: #9570 (comment)

[hoppity2]

@adrelanos Tagging Patrick as it's relevant, this is a Qubes-Whonix related issue.

[marmarek]

This is very much expected - if a qube's kernel is provided by dom0 (the default), kernel parameters are set in dom0 (kernelopts property). If qube wants to control its own kernel (including kernel options), there is the "(provided by qube)" option (that you found already).

@adrelanos would you like for Whonix to use kernel from within qube by default?
Related #5212

[hoppity2]

Ok, but as a user who knows security-misc is installed I would expect all of those hardening settings to apply. Also it would be good if all of those hardening settings did apply. This functionality should be documented, as it's not something the user would expect and people are lacking that extra security hardening.

Anyway a partial work around is to add those parameters into the Dom0 VM KernelOpts, though as noted not all of them can fit it, max 512 chars.

[adrelanos]

@adrelanos would you like for Whonix to use kernel from within qube by default? Related #5212

I would appreciate very much if #5212 was implemented and if Qubes in-VM kernel would become Qubes default (for Debian templates). However, only doing this for Qubes-Whonix might be troublesome.

This is problematic because it would derivate from Qubes Debian kernel default kernel.

This is unmaintainable, because if there are Qubes VM kernel specific crashes, issues, then these would be reported by users to Whonix where I would have no possibility to debug these except saying "please reproduce this in Qubes Debian and report the bug to Qubes".

[hoppity2]

Ok, thanks for the responses. I'm doubtful much progress would be made on #5212 anytime soon as the kernel is reliability critical.

@marmarek what would you suggest so I can enable all of the security-misc kernel boot time hardening options (I can manually workout and test to which ones work)? Again with my suggestion of increasing the length of the kernel opts VM attribute.

[marmarek]

I would appreciate very much if #5212 was implemented and if Qubes in-VM kernel would become Qubes default (for Debian templates).

That's not going to happen anytime soon because it's problematic for VMs with PCI devices (especially when using Debian kernel, due to old wifi drivers, but there are also other factors like significantly slower boot time in HVM mode). But those issues do not apply to Whonix, so it could be used there.
The main point of #5212 (simplifying setting it) is done already - it's a single option named kernel "provided by qube".
This option is regularly tested with both Fedora and Debian with all virt modes (this is how we know the HVM mode is problematic, but PVH works just fine). The test is disabled on Whonix but we can enable it there too (after ensuring the template(s) do have kernel and grub packages installed by default).

[adrelanos]

Given this excellent news to me...

@adrelanos would you like for Whonix to use kernel from within qube by default?

Yes, please. That would fix a lot of issues.

[marmarek]

Ok, since Whonix templates are based on minimal flavor, it normally doesn't include kernel and bootloader. So, this change proven to be a bit more complicated. Pull requests linked above should take care of it.