Qubes R4 after upgrades boot process stuck at "Reached Target Basic System"

[adrelanos]

Qubes OS version:

R4

Affected component(s):

dom0 boot

Steps to reproduce the behavior:

Install Qubes R4 on USB HDD. Install qubes-template-fedora-28. Upgrade all templates. Upgrade dom0. Reboot.

Expected behavior:

Normal boot.

Actual behavior:

Boot process stuck Reached Target Basic System.

General notes:

Already reinstalled, repeated the process to exclude a disk error but had the same result.

Related issues:

[marmarek]

Was the system rebooting normally after initial installation? Does it stuck before of after disk password prompt?
I guess only dom0 upgrade should matter here. Does it include testing repository, or only stable?

[adrelanos]

Was the system rebooting normally after initial installation?

Yes.

Does it stuck before of after disk password prompt?

After password prompt.

Does it include testing repository, or only stable?

Testing.

[adrelanos]

(I had Qubes R4 on USB HDD working great for months but some probably testing repository upgrade broke it. Using that as my testing system before I upgrade my main internal HDD Qubes R4 installation.)

[adrelanos]

How can I debug this? Problem is, I can't open a terminal at this point (ctrl + alt + F2).

Can I start recovery mode and then continue normal booting?

Can I boot normally but somehow get a terminal so we at least get a better error message or see where it is stuck?

[marmarek]

See https://freedesktop.org/wiki/Software/systemd/Debugging/, especially check systemd.debug-shell=1 on kernel command line.

[adrelanos]

Removed console=none, quiet and rhgb. Waited long enough. Ended up with a dracut:/# shell. Had a look at journalctl | less (or so). Got some error message now.

Check of pool qubes_dom0/pool100 failed (status:!). Manuel repair required!
Warning: dracut-initqueue timeout - starting timeout scripts
(many times)
Warning: Could not boot.
Warning: /dev/mapper/qubes_dom0-root does not exist
Warning: /dev/qubes_dom0/root does not exist

Does this give clues on the bug? Any further debug information useful?

[knopperx]

Hi. I'm having this same issue after running sudo qubes-dom0-update in dom0 terminal and rebooting. I have sensitive data on the system and would like to either fix the system or recover the data.

How do I manually repair qubes_dom0/pool100?

This seems to be an issue that has occurred for multiple people: https://groups.google.com/d/msg/qubes-users/PR3-ZbZXo_0/wem9wbGTBgAJ

[lunarthegrey]

@knopperx where you using the testing repository when you updated? What kernel are you running as well?

I'm not having this issue when using the stable repository. I plan to setup a test system in the future so I can help test issues like this.

[ryantate]

@adrelanos Did you ever get this resolved? Did you have to reinstall or was there some other solution? Hitting this myself: https://groups.google.com/forum/#!topic/qubes-users/u1gteTqGM3Y

[adrelanos]

Unfortunately, not. The only workaround I found was re-installing a newer version of Qubes which already had that offending upgraded package included.

[notbyosmosis]

All, I was able to recover from this state with a work-around today. Details here:

#5160

[adrelanos]

Potential duplicates / related:

• Qubes R4 after upgrades boot process stuck at "Reached Target Basic System" #4479 pool100
• Qubes post-Installation fails after reboot in qubes 4.0.2 #5160 pool100
• manual partitioning (custom installation): enhance documentation #5632 pool00
• Qubes post-Installation fails after reboot in qubes 4.0.2 #5160 pool00
• Upgrading from QubesOS 4.x doesn't result in the same end user configuration (discards and tpool) #5643 tpool

//cc @tlaurion

[github-actions]

This issue is being closed because:

• This issue is on the "Release 4.0 updates" milestone.
• Qubes OS 4.0 reached EOL (end-of-life) over one year ago.
• There has not been any activity on this issue in over one year.

If anyone believes that this issue should be reopened and reassigned to an active milestone, please leave a brief comment.
(For example, if a bug still affects Qubes OS 4.1, then the comment "Affects 4.1" will suffice.)