Migrate disk image building to systemd-repart

[DaanDeMeyer]

Let's switch to systemd-repart to build disk images. systemd-repart
allows building disk images declaratively. repart will assemble a
disk image according to a set of partition definition files, which
it will search for in a set of predefined locations or in the locations
provided via the --definitions option. Partition definition files are
defined in the usual systemd ini style and define a single partition
each. Many aspects of the partition can be configured, including its
size, type, and how it should be populated.

Of note should be the CopyFiles= setting that allows providing a set
of files/directories from which the partition should be populated. The
CopyFiles= setting takes a list of path pairs, where the first path in
a pair defines the source location, and the second path the destination
location in the partition. If combined with repart's --root or --image
options, the source path is resolved relative to the given root directory.

Along with switching to systemd-repart, we also rework how we build
images. Previously, we first provisioned the disk image with all its
partitions, then mounted the disk image and finally populated it with
contents. Now, we provision to a regular directory on the host filesystem
first, followed by creating the disk image with systemd-repart. The partition
definition files can be supplied via the --repart-directory option or
in the mkosi.repart/ directory. If not provided, a default configuration
consisting of an automatically sized root partition and an ESP partition of
256M is used.

repart's --root switch is used so that the partition definition files
don't need to encode the full path to the root directory of the image
we're building. For example, to copy the root directory to a root
partition, it's sufficient to specify "CopyFiles=/:/" in the definition
file.

If we're building non-bootable images, any ESP partition definitions
are excluded. To allow building UKIs with an embedded roothash cmdline
parameter, we run repart twice, first to populate all partitions except
the ESP/XBOOTLDR partitions, and again to populate the ESP/XBOOTLDR
partitions including any UKIs with embedded roothash cmdline parameter
(which we determined when running repart the first time).

Because we don't know up-front anymore where the ESP partition will be
mounted, all boot loader files are installed to /boot. So to populate
an ESP partition, you'd use "CopyFiles=/boot:/" in the partition
definition file of the ESP partition.

Also, since we don't know up-front anymore which filesystem we'll be
building for, we stop installing filesystem related packages by default.
Users will be required to add the necessary filesystem packages themselves.
Similarly, we also stop installing cryptsetup and device-mapper by default
since we don't know upfront anymore whether partitions will be verity
protected or encrypted.

By switching to systemd-repart, we also set ourselves up for building
images without needing root privileges or loop devices. systemd-repart
is fully capable (from v253 onwards) of building disk images without
needing root privileges or loop devices. After we switch to
systemd-repart, we'll only need root privileges to be able to run
systemd-nspawn (which should not be necessary in the future anymore).

This PR also removes all the option related to disk image building that
can now be specified in repart definition files instead. This includes:

• Format=gpt_xxx options are replaced with a single "disk" options.
  Filesystem can now be specified with repart's Format= option
• Format=plain_squashfs (Can be reproduced by a single repart squashfs
  root partition combined with SplitArtifacts=yes)
• Verity= (Replaced by repart's Verity= options)
• Encrypt= (Replaced by repart's Encrypt= option)
• RootSize=, HomeSize=, VarSize=, TmpSize=, ESPSize=, SwapSize=, SrvSize=
  (Replaced by repart's size options)
• UsrOnly= (replaced with CopyFiles=/:/usr in a usr partition definition)
• OutputSplitRoot=, OutputSplitVerity=, (Replaced by repart's SplitName= option)
• OutputSplitKernel= (UKI is now always written to its own output file)
• GPTFirstLBA (Removed, no equivalent in repart)
• ReadOnly= (Replaced by repart's ReadOnly= option per partition)
• Minimize= (Replaced by repart's Minimize= option per partition)
• CompressFs= (No equivalent in repart, can be replicated by replacing mkfs.
  in $PATH with a script that adds the necessary command line option)
• CompressOutput= is renamed to Compress=
• MkSquashfs= (Can be replaced with a script in $PATH that invokes
  the correct binary)

We also remove the WithoutUnifiedKernelImages= switch as building unified
kernel images is trivial and fast these days.

[DaanDeMeyer]

This also gets rid of the pytest integration tests we added a while ago. Given that we'll be moving to integrating more with build systems in the future, building and booting the image as part of the same integration test seems like the wrong way to go. Instead we should just build the images in one step in a build system such as meson and then have the integration test(s) depend on it. For now, I switched us to the same approach used in systemd which does the trick as well.

[DaanDeMeyer]

I also switched us from using temporary files and passing around Optional[BinaryIO] everywhere to a staging directory that all functions that produce output write into and look into for dependent files. This was necessary because systemd-repart produces one or more output files which we can't account for with the current model and because it simplifies everything quite a bit as well.

[keszybz]

• Format=plain_squashfs (Can be reproduced by a single repart squashfs
  root partition combined with SplitArtifacts=yes)

I find this problematic. I expect that it'll make building of initrds more expensive. But I think it's not a blocker… In the worst case we can always add some support to build such images directly again.

• CompressFs= (No equivalent in repart, can be replicated by replacing mkfs.
  in $PATH with a script that adds the necessary command line option)

I don't think the approach of requiring the user to create fake wrappers to pass options is acceptable in the long run. But it shouldn't be too hard to add support for passing compression options again.

CompressOutput= is renamed to Compress=

Can we please not do that. There's bound to be many different compression settings. No need to recreate the old issue that was once solved.

[keszybz]

I think we should merge this asap.

[DaanDeMeyer]

I don't think the approach of requiring the user to create fake wrappers to pass options is acceptable in the long run. But it shouldn't be too hard to add support for passing compression options again.

It's not a mkosi problem anymore after we merge this. It'll be a repart problem. If we want to do this slightly nicer we'll need to have repart read extra mkfs options from environment variables like we do in homed and such.

I find this problematic. I expect that it'll make building of initrds more expensive. But I think it's not a blocker… In the worst case we can always add some support to build such images directly again.

cpio building hasn't changed, we only remove the plain squashfs output format because the complexity to keep it is bigger than then the added benefit of supporting it. cpio and tar output formats are still supported as before. (Unless you're saying that you want squashfs initrds?)

Can we please not do that. There's bound to be many different compression settings. No need to recreate the old issue that was once solved.

Reverted this, we drop CompressFs and keep CompressOutput and remove Compress.

[DaanDeMeyer]

CI failures are unrelated, shall we merge this and fix any remaining issues in follow up PRs?

[behrmann]

Sounds good to me, one last thing I noticed.

[behrmann]

With the files moved this isn't necessary anymore, no?

[DaanDeMeyer]

Only changed .gitignore, let's merge