Stuck after reboot

[jknipper]

Description

From time to time we see servers stuck in boot process after reboot. This only happens in rare cases after an OS update was applied.

Impact

The server is stuck after reboot and needs to be rebooted manually a second time to bring it up.

Environment and steps to reproduce

1. it's virtual machines and we are using VMware as a hypervisor
2. automatic reboot after OS update
3. we couldn't reproduce the behavior yet and it happens very rarely

Expected behavior

The machine boots without interruption.

Additional information

From the attached screenshot of the server console it seems that the boot process got stuck while trying to mount sysroot.

This issue started some time ago and is hard to debug for us. Any suggestions how we could investigate further in this matter are greatly appreciated!

[tormath1]

Hello @jknipper, thanks for raising this. Are you able to tell us since when (which release) did you notice this issue? It seems to be an issue with the disk, do you have a specific configuration for your instances?

[jknipper]

From what I can see in our alerts/logging it started in July or August on a regular basis. We are running the stable release and are updating all instances shortly after a new release is published. It's hard to see if this is on the hypervisor side or an issue with flatcar. There was no real change on how the instances are provisioned, maybe on VMWare side, I'll try to find out.

[tormath1]

All the instances reboot at the same time on the hypervisor?

[jknipper]

All the instances reboot at the same time on the hypervisor?

No, reboots are managed by some operator in Kubernetes and are pretty much random.

What is interesting from that screenshot is that the filesystem check for that same device seem so succeed but the mounting afterwards fails. When looking at the release history of Flatcar, our observations seem to collide with the switch from kernel version 6.1.x to 6.6.x. but could also be a coincidence.