fails to boot in CoreOS from git master

[cgwalters]

Something went wrong with the recent PR #2261 - that seems to have broken the scenario of overlaying latest git on latest FCOS. The symptom is systemd-sysusers.service failing, presumably before ostree-remount.service runs.

I haven't yet figured out why our main CI isn't failing.

[cgwalters]

I initially tried the obvious Before=systemd-sysusers.service in ostree-remount.service but:

[    5.434218] systemd[1]: local-fs-pre.target: Found ordering cycle on systemd-tmpfiles-setup-dev.service/start
[    5.434359] systemd[1]: local-fs-pre.target: Found dependency on systemd-sysusers.service/start
[    5.435955] systemd[1]: local-fs-pre.target: Found dependency on ostree-remount.service/start
[    5.437120] systemd[1]: local-fs-pre.target: Found dependency on var.mount/start
[    5.439209] systemd[1]: local-fs-pre.target: Found dependency on local-fs-pre.target/start
[    5.441340] systemd[1]: local-fs-pre.target: Job systemd-tmpfiles-setup-dev.service/start deleted to break ordering cycle starting with local-fs-pre.target/start

And anyways we really want #2187

[cgwalters]

Ugh...the mystery deepens here. If I just inject the new ostree-prepare-root into the initramfs via cosa buildinitramfs-fast, we're fine (as expected, there were no changes to anything ostree-in-initramfs since 2020.8. If I use cosa build-fast to just inject new userspace (but that doesn't change the initramfs).

But if I use the full make install DESTDIR=/srv/walters/builds/fcos/overrides/rootfs/ I...wait it's working now? Argh 😕

[cgwalters]

Ah! I think I figured out what happened. The glib-rs bindings merged into gtk-rs so I recently installed the build deps for those in my toolbox. Those dependencies include avahi-devel - and we don't include the ostree-avahi functionality in our Fedora ostree builds today. But when I rebuilt from source the local build auto-added the dependency. And I think this caused the ostree-system-generator binary to fail, which caused /var not to mount.

When debugging this at some point I used makesudoinstall on the host container, which caused cosa build supermin to fail. At that point I rebuilt --without-avahi which indirectly fixed the problem.

Also: of course what made this extra confusing is I tried to bisect this by building the v2020.8 release from source and still hit the problem.