newest beta 3185.1.0 does not boot #710
Comments
|
Probably related to flatcar/init#65 |
|
masking the service before rebooting solves the problem; however, knowing you HAVE to do that (across all my VMs) is a whole other story. |
|
Hi @goochjj, First thanks a lot for running beta nodes - do you have any log / information to provide for this:
|
|
Since my system is complete non-functional when I reboot, I have nothing. Now that I have the unit masked I can try to collect some logs |
|
Awesome, thanks. In the meantime, we are going to try to reproduce the issue - just to confirm: you were on a Stable and you switched to Beta ? |
|
Beta to Beta and Stable to Beta both do it |
|
It may be part of some of my customizations, i.e.: systemctl cat ensure-sysext.service/usr/lib/systemd/system/ensure-sysext.service[Unit] /etc/systemd/system/service.d/email-failure.conf[Unit] systemctl start ensure-sysextA dependency job for ensure-sysext.service failed. See 'journalctl -xe' for details. Apr 07 15:31:33 NetFlyTrap.redacted bashcmd(root|0)[2310]: /root# systemctl start ensure-sysext |
|
Removing my onfailure entry doesn't help |
|
Ok, thanks for the logs. I tried to reproduce locally on QEMU environment from What does |
|
I've been able to reproduce with the following ignition: {
"ignition": {
"version": "3.3.0"
},
"passwd": {
"users": [
{
"name": "core",
"sshAuthorizedKeys": [
"ssh-rsa ..."
]
}
]
},
"systemd": {
"units": [
{
"dropins": [
{
"contents": "[Unit]\nOnFailure=echo@%n.service\n",
"name": "failure.conf"
}
],
"name": "ensure-sysext.service"
},
{
"contents": "[Unit]\nDescription=Show error\n[Service]\nType=fork\nExecStart=/usr/bin/echo error\n[Install]\nWantedBy=multi-user.target\n",
"name": "[email protected]"
}
]
}
}I think the |
|
failuremgmt basically does this [Service] Which just emails me :-D |
|
Status looks ok Apr 07 16:28:33 NetFlyTrap.redacted systemd[1]: Started Unit Status Failure Management. |
|
The part I can't really test is a dependency on sysinit - to get it to boot I have to mask the ensure-sysext service, which I think is probably invalidating some of my testing, because it's entirely possible something just isn't there yet when it's running in realtime. And since I've now definitely booted this VM into the new OS, it'd be hard to back out I suppose. I can see about snapshotting this VM so I can recover quicker. |
|
More logs mined from journalctl |
|
Confirmed working without my OnFailure= I will look into other options, unless you have some suggestions. Really I have the email notices in there to detect boot-up issues, but I'd love for that dropin to only apply after sysinit... or at a specific point in the boot sequence. I'll have to dig further and come up with a solution. For that matter if sysext is firing, /usr and /opt may not be completely consistent yet - so ... it's really not appropriate for my drop-in to be using them that early in the boot. I'm not sure that occurred to me until right now, it's just always worked. Thanks! |
|
I've made a unit that fires after network-online.target that dumps my service.d/ dropin into /run, instead of having it in etc. This seems to work quite nicely, actually, and eliminates the problem. I'll leave it to you whether you want to investigate further. :-D |
|
Seems we need to duplicate the conditions from |
The unit failed to execute when the dependency systemd-sysext.service was skipped because it's not needed. Normally this is harmless but it still triggers a OnFailure event as reported in flatcar/Flatcar#710 Use the same logic for skipping execution as done in systemd-sysext.service - we can't use "systemctl is-enabled" because Flatcar's inbuilt unit symlinks are not reported as "enabled".
This pulls in flatcar/init#68 to skip the ensure-sysext unit when systemd-sysext is skipped to prevent a dependency failure being reported. Closes: flatcar/Flatcar#710
|
This will be a stupid questions but could the ensure-sysext.service commands be added via drop-in to systemd-systext.service as ExecStartPost? |
Seems also to be a good option, I've tested it and it works well even if other ExecStartPost entries are present. |
|
(Not sure how a user would disable a drop-in, maybe with an empty one of the same name) |
|
Oh, if disabling is a concern then a separate unit is probably more sane. |
|
Not really the main concern, the main concern was to make clear that this is something custom to Flatcar. The ExecStartPost action is nice because it's something we could try to upstream but I'm not really sure it gets accepted… I'll take a note. |
This pulls in flatcar/init#68 to skip the ensure-sysext unit when systemd-sysext is skipped to prevent a dependency failure being reported. Closes: flatcar/Flatcar#710
This pulls in flatcar/init#68 to skip the ensure-sysext unit when systemd-sysext is skipped to prevent a dependency failure being reported. Closes: flatcar/Flatcar#710
This pulls in flatcar/init#68 to skip the ensure-sysext unit when systemd-sysext is skipped to prevent a dependency failure being reported. Closes: flatcar/Flatcar#710
|
Will be resolved in the next bug fix releases for Alpha/Beta |
pothos
moved this to Upcoming / Backlog
in Flatcar tactical, release planning, and roadmap
pothos
moved this from Upcoming / Backlog
to Implemented
in Flatcar tactical, release planning, and roadmap
Description
Dependency failed for ensure-sysext.service (over and over and over and over again until I reboot)
Impact
Machines do not reboot cleanly after the update is applied. I have to reboot the VM manually (which boots on the old OS image)
Environment and steps to reproduce
Expected behavior
Doesn't fail.
The text was updated successfully, but these errors were encountered: