ignition-kargs.service racing with other units for access to /boot

[dustymabe]

In the case that I'm seeing ignition-kargs.service, which is accessing /boot read-write is racing with coreos-copy-firstboot-network which accesses boot read-only.

[    2.625189] ignition[533]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"                                                             
[    2.626376] ignition[533]: kargs: op(1): [started]  updating kernel arguments                                                                    
[    2.630045] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.                                           
[    2.631751] ignition[533]: kargs: op(1): [failed]   updating kernel arguments: exit status 1: Cmd: "ignition-kargs-helper" "--should-exist" "ip=192.168.122.112::192.168.122.1:255.255.255.0:initrdhost:ens2:none:208.67.222.222" "--should-exist" "rd.neednet=1" Stdout: "" Stderr: "Error: mounting 
device /dev/disk/by-label/boot on /tmp/coreos-installer-2gYC0L\n\nCaused by:\n    EBUSY: Device or resource busy\n"                                 
[    2.634541] coreos-copy-firstboot-network[532]: info: no files to copy from /mnt/boot_partition/coreos-firstboot-network. skipping

I'm able to workaround for now with something like:

$ git diff overlay.d/05core/usr/lib/dracut/modules.d/35coreos-network/coreos-copy-firstboot-network.service 
diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/35coreos-network/coreos-copy-firstboot-network.service b/overlay.d/05core/usr/lib/dracut/modules.d/35coreos-network/coreos-copy-firstboot-network.service
index 6ba3966..3a997ce 100644
--- a/overlay.d/05core/usr/lib/dracut/modules.d/35coreos-network/coreos-copy-firstboot-network.service
+++ b/overlay.d/05core/usr/lib/dracut/modules.d/35coreos-network/coreos-copy-firstboot-network.service
@@ -47,6 +47,7 @@ After=coreos-multipath-wait.target
 # hook which will generate NM configs from the network kargs, but we want to
 # have precedence.
 After=coreos-enable-network.service
+Before=ignition-kargs.service
 
 [Service]
 Type=oneshot

But I wonder if there's a more complete solution.

I might be hitting this now when no one else has because I'm testing out newer dracut over in coreos/fedora-coreos-config#1068

[jlebon]

I think adding stronger ordering in that unit makes sense, though I would probably go with After=ignition-kargs.service instead. There's no point to do that work if we're going to reboot anyway.

[dustymabe]

though I would probably go with After=ignition-kargs.service instead.

Yeah. That's what I originally did, but I ended up with a systemd unit ordering loop when testing with dracutdevs/dracut#1548

[dustymabe]

though I would probably go with After=ignition-kargs.service instead.

Yeah. That's what I originally did, but I ended up with a systemd unit ordering loop when testing with dracutdevs/dracut#1548

Here's the ordering cycle:

[    1.096580] systemd[1]: ignition-fetch.service: Found ordering cycle on network.target/start                       
[    1.097192] systemd[1]: ignition-fetch.service: Found dependency on nm-initrd.service/start                        
[    1.097794] systemd[1]: ignition-fetch.service: Found dependency on coreos-copy-firstboot-network.service/start                                    
[    1.098521] systemd[1]: ignition-fetch.service: Found dependency on ignition-kargs.service/start                                                   
[    1.099165] systemd[1]: ignition-fetch.service: Found dependency on ignition-fetch.service/start                   
[    1.099807] systemd[1]: ignition-fetch.service: Job network.target/start deleted to break ordering cycle starting with ignition-fetch.service/start

[dustymabe]

I'm currently not able to reproduce the race failure (after all, it is a race). Should I continue to add in the proposed patch of Before=ignition-kargs.service or should we wait til we see this again?

[dustymabe]

Should I continue to add in the proposed patch of Before=ignition-kargs.service or should we wait til we see this again?

added it as a commit in coreos/fedora-coreos-config#1068