Simplify and promote using in-vm kernel #5212
Comments
marmarek
added
T: enhancement
marmarek
added
the
release notes
|
Peanut gallery has arrived. :) How would Qubes dom0 be safely informed what bootloader/kernel/initrd combo is installed/available inside PV/PVH VMs? Or is there an agreed upon naming standard that is assumed? So...Qubes already uses a notation of :object elsewhere in Qubes (e.g. devices). Therefore, perhaps consider something along the lines of: Perhaps greying out items that aren't applicable to the current VM type? Naming strategy allows future possibility of attempting to boot VM1 off the bootloader/kernel/initrd locally stored with VM2, if that would ever be useful (possibly just a crazy thought). For kernel options, store per-VM changes from the default per selectable kernel, nothing if stored if the default value is left as is? Brendan |
The whole idea is it wont. It will leave it to a bootloader within the VM itself, using predetermined protocol. In case of HVM, it's loading it from MBR (in the future there may be an option for UEFI). In case of PV/PVH, it's launching grub2, which will load a config from /boot within a VM. |
|
Will there still be a (non-default) Qubes provided VM kernel? I guess I should look into upstreaming 0006-block-add-no_part_scan-module-parameter.patch (used by Split dm-crypt)... |
I don't plan to. But there will be still (non-default) dom0-provided kernel. |
You mean, built by Fedora? |
|
No, same as now. |
|
Oh, I see. By "VM kernel", I meant a kernel used by the VM but located in dom0. Sorry for the confusion. |
|
In-VM kernel all the way. Seems much more expected from all parties. (If Qubes didn't introduce dom0 kernel, I'd never expect it.) (Parties include users, developers, applications by distribution, distribution maintainers, grub.d.) I'd go as far as making dom0 kernel user opt-in only.
This is one of the strongest points for dom0 I have to acknowledge even though I strongly favor In-VM kernel. For any supported template, this would be better handled by distribution specific kernel fixes inside the VM by Qubes repository? You might say this is hard? Might be good point. But... Well, if you promote in-vm kernel, then "half" users will have one thing, and half the other thing. What you tell them then? "We have a security issue, therefore go back to dom0 provided kernel." Double maintenance effort. |
|
It's already possible to test and use in-vm kernel, even for PVH. If you want to help make it default @adrelanos , here are testing steps for PVH:
It should "just work" for Debian 10 and result in a working AppArmor and other features. But require more testing. In case of Whonix, which is based on Debian 10 minimal, it might require installing Note it won't work out of the box on Fedora 30, because of BLS mentioned in issue description. It might be enough to set |
|
In case of Whonix, which is based on Debian 10 minimal, it might require installing `qubes-kernel-vm-support` if not already there.
Package qubes-whonix-shared-packages-recommended `Depends:` on
qubes-kernel-vm-support in Whonix 15. Not something Whonix should
interfere with specifically (no strong rationale at the moment). I
thought it is a good idea to be on par with Qubes Debian templates and
kinda hoped for this change too.
|
|
working:
```
[user@dom0 ~]$ sudo xl console debian-10
.[30m.[47mWelcome to GRUB!
.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H [ grub-xen.cfg 424B 100%
1.57KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.e
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1H.[1;36H [ grub.cfg 5.43KiB 100% 7.97KiB/s
].[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[H.[J.[1;1H Booting `Debian GNU/Linux'
Loading Linux 4.19.0-5-amd64 ...
.[4;24H [ vmlinuz-4.19.0-5-amd 4.98MiB 100% 4.96MiB/s
].[4;1HLoading initial ramdisk ...
.[5;22H [ initrd.img-4.19.0-5- 31.95MiB 100% 22.11MiB/s ].[5;1H[
0.000000] Xen Platform PCI: unrecognised magic value
[ 0.193469] ACPI: No IOAPIC entries present
```
Not working:
```
[user@dom0 ~]$ sudo xl console debian-10
.[30m.[47mWelcome to GRUB!
.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H [ grub-xen.cfg 424B 100%
1.57KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.e
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1H.[1;36H [ grub.cfg 5.43KiB 100% 7.97KiB/s
].[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[H.[J.[1;1H Booting `Debian GNU/Linux'
Loading Linux 4.19.0-5-amd64 ...
.[4;24H [ vmlinuz-4.19.0-5-amd 4.98MiB 100% 4.96MiB/s
].[4;1HLoading initial ramdisk ...
.[5;22H [ initrd.img-4.19.0-5- 31.95MiB 100% 22.11MiB/s ].[5;1H[
0.000000] Xen Platform PCI: unrecognised magic value
[ 0.193469] ACPI: No IOAPIC entries present
```
|
|
Some issues.
Suggest to automate in existing (and new) template VMs on upgrade:
sudo mkdir /boot/grub
sudo update-grub2
https://www.qubes-os.org/doc/managing-vm-kernel/ does not mention
installing grub inside VMs.
Whonix VMs has qubes-kernel-vm-support package installed but lacks
/usr/sbin/update-grub2. So the "right" grub package to be installed.
sudo apt install --no-install-recommends grub2
user@host:~$ sudo mkdir /boot
mkdir: cannot create directory ‘/boot’: File exists
user@host:~$ sudo ls -la /boot
ls: cannot open directory '/boot': No such device
user@host:~$ sudo mkdir /boot/grub
mkdir: cannot create directory ‘/boot/grub’: No such device
user@host:~$ mount | grep boot
systemd-1 on /boot type autofs
(rw,relatime,fd=42,pgrp=1,timeout=120,minproto=5,maxproto=5,direct,pipe_ino=11581)
And do we need "sudo apt install --no-install-recommends os-prober"?
(grub2 recommended package.)
|
|
This happened in |
|
This messed up graphics inside the VM. After VM reboot, maximizing terminal emulator window triggered this error again. Installation of anbox may or may not be the actual cause. Didn't test VM kernel much before. |
grub itself isn't necessary. Just grub config. Which means a tool to generate it.
Oh, so
what? I don't have anything like this. Do you have /boot in |
I recommend against. |
This is weird. It's this one: https://github.com/torvalds/linux/blob/33920f1ec5bf47c5c0a1d2113989bdd9dfb3fae9/mm/memory.c#L2013 (page passed to u2mfn is a "huge page") |
|
Again (previously reported in #5212 (comment)) debian-buster (manually updated debian stretch to debain buster) (not debian-10) TemplateVM So cannot reproduce boot being mounted in unexpected ways. Does not boot with dom0 setting Is there any debug information that I could provide? |
|
Marek Marczykowski-Górecki:
> Aug 07 14:55:51 debian-10 kernel: kernel BUG at mm/memory.c:2201!
This is weird. It's this one: https://github.com/torvalds/linux/blob/33920f1ec5bf47c5c0a1d2113989bdd9dfb3fae9/mm/memory.c#L2013
I don't see why it would crash this way when using Debian kernel, but not our Qubes one. Maybe something about kernel config?
It's pretty much a stock TemplateVM (debian-10) that I did nothing with
except upgrading. No custom compiled kernel, no security-misc package,
nothing special.
|
Try removing Also, something went wrong with kernel options, as some are duplicated. But I don't think that cause the problem. |
|
Same output without For comparison debian-10 TemplateVM. And debian-10 TemplateVM without |
|
Some boot parameter by package https://github.com/Whonix/security-misc is causing this. After removing all of these, VM boots. Any idea which one, and why? The other bug (kernel crash, making VM unusable) |
|
Command Where would be a good place to do that? |
|
Can |
|
Note to self: Root device
|
Automatically translate it to appropriate pvgrub flavor. QubesOS/qubes-issues#5212
Initramfs at 22MB is small enough to start in PVH mode, but too big to start in HVM mode. Exclude tpm2-tss dracut module to save another 5MB, which makes it work in HVM mode too. QubesOS/qubes-issues#8540 QubesOS/qubes-issues#5212
Initramfs at 22MB is small enough to start in PVH mode, but too big to start in HVM mode. Exclude tpm2-tss dracut module to save another 5MB, which makes it work in HVM mode too. QubesOS/qubes-issues#8540 QubesOS/qubes-issues#5212 (cherry picked from commit b360c4a)
Nowadays empty kernel is automatically converted to pvgrub2, so don't refuse that value. Fixes: ba8bc65 "Allow setting "none" kernel to use in-vm kernel in any virt_mode" QubesOS/qubes-issues#5212
"None" kernel gets translated to appropriate grub variant according to virt_mode. QubesOS/qubes-issues#5212
This makes sense for testing if in-vm kernel works out of the box in selected templates. QubesOS/qubes-issues#5212
This makes sense for testing if in-vm kernel works out of the box in selected templates. QubesOS/qubes-issues#5212
This makes sense for testing if in-vm kernel works out of the box in selected templates. QubesOS/qubes-issues#5212
Nowadays empty kernel is automatically converted to pvgrub2, so don't refuse that value. Fixes: ba8bc65 "Allow setting "none" kernel to use in-vm kernel in any virt_mode" QubesOS/qubes-issues#5212 (cherry picked from commit 9498dfe)
"None" kernel gets translated to appropriate grub variant according to virt_mode. QubesOS/qubes-issues#5212 (cherry picked from commit 9f3bdb4)
The problem you're addressing (if any)
Currently by default all qubes use kernel provided by dom0. This has multiple issues:
Using in-vm kernel isn't enabled by default and it's not straightforward:
Describe the solution you'd like
Unify setting in-vm kernel - possibly translate "(none)" to "pvgrub2" if virt_mode is PV and to "pvgrub2-pvh" if virt_mode is PVH. Note that "(none)" normally is an invalid choice for PV/PVH.
This would require adjusting "(none)" label at UI level, to be less confusing/magic. Something like "(use kernel from within the qube)".
Set is as default value.
Where is the value to a user, and who might that user be?
Less deviation from template's system, according to https://www.qubes-os.org/faq/#what-is-qubes-attitude-toward-changing-guest-distros
Possible drawbacks
This change may lead also to some issues:
Describe alternatives you've considered
Implement automatic pvgrub choice for kernel "(none)", but don't set it as default.
Relevant documentation you've consulted
https://www.qubes-os.org/doc/managing-vm-kernel/
The text was updated successfully, but these errors were encountered: