Roadmap: systemd-sysext and systemd-confext

[pothos]

We have started to point users at systemd-sysext for bringing their software and configuration to Flatcar as overlay on /usr.
To get there, a few workaround were needed. While there were discussions with upstream how to solve this, the work is yet to be done. The first step would be to drop our customizations through some upstream and Flatcar improvements. This should already allow us to transition to sysext as torcx replacement and for the OEM tools. Then we can look into unlocking more use cases for systemd-sysext and systemd-syscfg after doing the needed upstream development. Finally we can look at some syscfg use cases in Flatcar.

1. Drop our customizations to reach a state where there is one consistent future-proof sysext behavior supported by upstream:

• Make upstream systemd-sysext reload the units by default: systemd-sysext refresh should also issue a daemon-reload as done by systemctl enable X, this allows us to drop this line https://github.com/flatcar/init/blob/fbb1ec8e9d95598d65e4eac408dbc5f75b9b6138/systemd/system/ensure-sysext.service#L14: Make upstream systemd-sysext reload the units by default #984
• Move OEM mount point out of /usr #766 which allows us to drop https://github.com/flatcar/init/blob/flatcar-master/systemd/system/systemd-sysext.service.d/10-mount-oem-partition.conf

With the above we can complete the transition of OEM tools and torcx to sysext.

2. Make sysext/syscfg work for more cases like kernel module settings or systemd configuration itself:

• Run systemd-sysext for /sysroot in the initrd already, this can be tested in Flatcar (should run after initrd-setup-root-after-ignition.service) but this new unit should be upstreamed and as far as I know the upstream wish was also to support mounting /var through a kernel cmdline parameter to be able to account for sysext images under /var but not sure if this is a blocker: Setup systemd-sysext for the rootfs in the initrd already #985
• Support a mutable mode for systemd-sysext and systemd-syscfg. For systemd-sysext this mutable mode will help traditional distro users on Debian or Fedora become accustomed to systemd-sysext. For systemd-syscfg we could use this mutable mode for our /etc overlay mount, making /usr/share/flatcar/etc a syscfg directory and setting up a symlink to it from /usr/lib/syscfgs/. Support a mutable mode for systemd-sysext/confext #986
• Make systemd-sysext replace the overlay mount atomically with Linux 6.5 "mount beneath", and then unmounting the old one with a regular lazy umount: Atomic systemd-sysext remount #1328
• Add extension-release file setting to restart a list of given services (needed even with the daemon reload): Restart option in systemd-sysext's extension-release file #1329
• As ergonomic improvement, systemd should support UpheldBy in the [Install] section and a TARGET.upholds dir with symlinks: Support UpheldBy as systemd property for enabling units #987
• Add systemd-sysext refresh --check-conflicts flag that prevents loading extensions when one extension would either shadow the base OS or another extension

This makes sysext/syscfg feature complete for advanced use and should also help to promote it outside of Flatcar.

3. Future ideas to use systemd-syscfg in Flatcar:

• Maybe leverage systemd-syscfg for OEM files that need to be under /etc - see use sysext in azure OEM images flatcar-archive/coreos-overlay#2506 (comment) for a case where we would copy a file to /etc from a service and the file won't be updated → or better fix the lack of support for /usr for vendor-supplied configs if we encounter it!
• Use ignition-apply to bake a syscfg image, then one could replace an old syscfg with a newer one for live config updates without a reboot, given that, e.g., a small helper allows to embed the ignition-apply-baked syscfg in a final Ignition config to be used on first boot

[pothos]

Question from @tormath1: Exact behavior and drawbacks of Upholds= for the multi-user.target

Edit:
Answer: The target still gets reached if the service it should uphold is always failing or doesn't exist.