nixos-rebuild hangs when building/switching system

[crabdancing]

Describe the bug

I'm experiencing a bug where, every time I try to do nixos-rebuild switch, nixos-rebuild boot` or similar, it responds with the following:

Failed to start transient service unit: Unit nixos-rebuild-switch-to-configuration.service was already loaded or has a fragment file.

It then hangs indefinitely.

I'm not exactly sure how to reproduce this, as I have a complex multi-repo flake, and furthermore, the results don't seem to be consistent. This issue crops up on exactly one of my servers, but none of the 15 or so other Nix systems I maintain with the same multi-system flake. Which makes me think it's some stale piece of state.

Expected behavior

I expect to be able to rebuild the system configuration without hanging indefinitely or error messages.

Additional context

Given the system in question is one of my build servers, I rarely if ever do garbage collection, since I like to keep old versions for reference.

Notify maintainers

@roberth
@thiagokokada

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 6.8.11, NixOS, 24.05 (Uakari), 24.05.20240531.805a384`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - nixpkgs: `/etc/nixPath`

---

Add a 👍 reaction to issues you find important.

[crabdancing]

After running nixos-rebuild switch --repair for a long while, it finally completed, and the problem has solved itself. Maybe a result of some sort of corruption of the Nix store? Hopefully anyone googling finds this to be a useful hint if you run into this one.

It should also be noted that --repair is not present on the man page for nixos-rebuild -- perhaps because it was removed and then later re-added?

[eclairevoyant]

--repair is not present on the man page for nixos-rebuild

should be fixed by #316301

[n8henrie]

@crabdancing how long did --repair take? I started getting this error yesterday, and it's been trying to --repair for hours and hours...

[crabdancing]

I'm not really sure exactly how long you have to run it, but expect it to take a very, very long time -- since it has to hash and check every single package in the Nix store.

On the machine this happened on, it had an AMD Ryzen 7 5700X & a fast NVME boot drive -- and it ended up taking probably around 4 hours. I expect on lesser hardware it could take literally days, depending on how big your system configuration is.

My recommendation is generally to try just leaving it and being patient, and see if the damn thing can eventually fix itself.

[n8henrie]

Interesting. I tried running nix store verify on all of /etc/systemd/system/**/*.service | grep -v '/dev/null' and found no errors.

Eventually I just tried sudo systemctl stop nixos-rebuild-switch-to-configuration.service and the nixos-rebuild switch immediately succeeded. ¯\_(ツ)_/¯

[crabdancing]

Eventually I just tried sudo systemctl stop nixos-rebuild-switch-to-configuration.service and the nixos-rebuild switch immediately succeeded. ¯_(ツ)_/¯

Yeah, that happened in my experience too. Only, it didn't fix anything long term for me. Maybe it was just corrupted in secondary memory, but not in your store?