systemd-boot not installing on unstable #156121

colemickens · 2022-01-22T02:02:17Z

Describe the bug

I'm not sure if I'm doing something silly, but my /boot was full, so I figured I'd empty it out, set boot.loader.systemd-boot.configurationLimit and then re-deploy.

However, this didn't go well (I hope the machine doesn't restart!):

+ ssh localhost 'echo "/nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110" | env CACHIX_SIGNING_KEY="eIuqcXYgSpe2wktg7fiSHuxo2Cko2lTEgxHxg2bl3YJs2snoV8xEQHht
3gE6MQ3zk2K7xVw91B7gXgrFWxv8jg==" cachix push colemickens'
++ printf '\"%s\" ' sudo nix build --no-link --profile /nix/var/nix/profiles/system /nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110
+ ssh 100.103.91.27 '"sudo" "nix" "build" "--no-link" "--profile" "/nix/var/nix/profiles/system" "/nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110" '
warning: unknown experimental feature 'ca-references'
++ printf '\"%s\" ' sudo nix shell -vv /nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110 -c switch-to-configuration switch
+ ssh 100.103.91.27 '"sudo" "nix" "shell" "-vv" "/nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110" "-c" "switch-to-configuration" "switch" '
All done.
warning: unknown experimental feature 'ca-references'
systemd-boot not installed in ESP.
No default/fallback boot loader installed in ESP.
Failed to open "/boot/EFI/systemd/.#systemd-bootx64.efi354fbb7c5fcf5579" for writing: No such file or directory
Failed to open "/boot/EFI/BOOT/.#BOOTX64.EFI145e91ccf64bc344" for writing: No such file or directory
could not find any previously installed systemd-boot, installing.
Traceback (most recent call last):
  File "/nix/store/f8gnqlqvsp6yhahaf0alkmqb0k4j8igb-systemd-boot", line 305, in <module>
    main()
  File "/nix/store/f8gnqlqvsp6yhahaf0alkmqb0k4j8igb-systemd-boot", line 257, in main
    subprocess.check_call(["/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/bin/bootctl", "--path=/boot", "update"])
  File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/bin/bootctl', '--path=/boot', 'update']' returned non-zero exit status 1.

So, let's remote in and try to activate the generation with NIXOS_INSTALL_BOOTLOADER set:

cole@jeffhyper ~
❯ sudo env NIXOS_INSTALL_BOOLOADER=1 "/nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110/bin/switch-to-configuration" switch
systemd-boot not installed in ESP.
No default/fallback boot loader installed in ESP.
could not find any previously installed systemd-boot, installing.
Failed to open "/boot/EFI/systemd/.#systemd-bootx64.efi5089e392206a681c" for writing: No such file or directory
Failed to open "/boot/EFI/BOOT/.#BOOTX64.EFI87290d639679f51f" for writing: No such file or directory
Traceback (most recent call last):
  File "/nix/store/f8gnqlqvsp6yhahaf0alkmqb0k4j8igb-systemd-boot", line 305, in <module>
    main()
  File "/nix/store/f8gnqlqvsp6yhahaf0alkmqb0k4j8igb-systemd-boot", line 257, in main
    subprocess.check_call(["/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/bin/bootctl", "--path=/boot", "update"])
  File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/bin/bootctl', '--path=/boot', 'update']' returned non-zero exit status 1.

Steps To Reproduce

Steps to reproduce the behavior:

Use current nixos-unstable:
These are my boot settings:

{ 
#... 
  boot = {
      loader.systemd-boot.enable = true;
      loader.systemd-boot.configurationLimit = 3;
      loader.efi.canTouchEfiVariables = true;
  };
}

Expected behavior

systemd-boot to be (re-)installed correctly.

The text was updated successfully, but these errors were encountered:

colemickens · 2022-01-22T02:03:06Z

Oooof, this fixes:

cole@jeffhyper ~
❯ sudo mkdir -p /boot/EFI/{systemd,BOOT}

cole@jeffhyper ~
❯ sudo env NIXOS_INSTALL_BOOLOADER=1 "/nix/store/4i923igvgj0r29cif8rsgaxn89x9j4h9-nixos-system-jeffhyper-22.05.20220120.bb27110/bin/switch-to-configuration" switch
systemd-boot not installed in ESP.
No default/fallback boot loader installed in ESP.
could not find any previously installed systemd-boot, installing.
Copied "/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/nix/store/cfhpzlarbhfw3scj91dcz5ai04ayfzik-systemd-249.7/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
warning: unknown experimental feature 'ca-references'
stopping the following units: hv-kvp.service, hv-vss.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-udevd.service, tailscaled.service
NOT restarting the following changed units: systemd-fsck@dev-disk-by\x2dlabel-BOOT.service
activating the configuration...
setting up /etc...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint 9800b3ce7c1c4ccfbd09095bd4ef0760fad2a89d
reloading user units for cole...
setting up tmpfiles
reloading the following units: dbus.service
restarting the following units: home-manager-cole.service, sshd.service
starting the following units: hv-kvp.service, hv-vss.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, tailscaled.service

colemickens · 2022-01-22T03:01:02Z

Observation, it says:
could not find any previously installed systemd-boot, installing.

but then runs bootctl update instead of bootctl install (not sure if update is meant to be idempotent and usable as install or not.... ?)

colemickens · 2022-01-22T03:09:34Z

Is this an upstream systemd-boot change that causes install to not mkdir -p ? AFAICT, even with recent changes to nixpkgs->nixos systemd loader, it should be triggering the bootctl install path since I'm setting NIXOS_INSTALL_BOOTLOADER.

totoroot · 2022-02-02T10:57:09Z

I can confirm as this has happened on my NixOS 21.11 install as well. I changed my hardware configuration to use another disk for boot and ultimately reverted the changes.

I couldn't reinstall/update the system due to it throwing the mentioned error

systemd-boot not installed in ESP.
No default/fallback boot loader installed in ESP.
could not find any previously installed systemd-boot, installing.

Creating the directories like @colemickens suggested actually solved it, but still a strange issue.

sudo mkdir -p /boot/EFI/{systemd,BOOT}

jonringer · 2022-02-02T18:08:33Z

first thought is to do bootctl install instead of bootctl update when "needs_install" is true;

nixpkgs/nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py

Lines 256 to 260 in ee4d3df

    
               if needs_install: 
        
                   subprocess.check_call(["@systemd@/bin/bootctl", "--path=@efiSysMountPoint@", "update"]) 
        
           mkdir_p("@efiSysMountPoint@/efi/nixos") 
        
           mkdir_p("@efiSysMountPoint@/loader/entries")

YellowOnion · 2022-02-13T06:47:01Z

work around isn't working with nixos-install (or this is another similar issue).

We really need better test coverage, if stuff like this breaks the installer, it should be easy to catch.

installing the boot loader...
setting up /etc...
Running in chroot, ignoring request.
Copied "/nix/store/24ljibki63lxk0m11qnw8fh9smh64g3x-systemd-249.7/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/nix/store/24ljibki63lxk0m11qnw8fh9smh64g3x-systemd-249.7/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
Random seed file /boot/loader/random-seed successfully written (512 bytes).
Created EFI boot entry "Linux Boot Manager".
mktemp: failed to create file via template ‘/mnt/tmp.p7wN2peYyL/tmp.XXXXXXXXXX’: No such file or directory
Traceback (most recent call last):
  File "/nix/store/g66xbwwxlxbxnfq6i1zzdc2nbqydsw28-systemd-boot", line 305, in <module>
    main()
  File "/nix/store/g66xbwwxlxbxnfq6i1zzdc2nbqydsw28-systemd-boot", line 293, in main
    subprocess.check_call("/nix/store/si4frm8y336b00animvm77mkmqmc3k9d-copy-extra-files")
  File "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/nix/store/si4frm8y336b00animvm77mkmqmc3k9d-copy-extra-files' returned non-zero exit status 1.

Atemu · 2022-11-17T19:06:08Z

This is still an issue. You can easily reproduce it by clearing /boot and re-activating your current generation.

Why aren't our installer tests running into this aswell when we do on our real systems?

Atemu · 2022-11-17T19:49:59Z

Ah, this issue is about nixos-install. I am experiencing this issue with nixos-rebuild.

ncfavier · 2023-01-01T14:37:59Z

Was this fixed by #172849? I can't reproduce this issue, rm -rf /boot/*; NIXOS_INSTALL_BOOTLOADER=1 nixos-rebuild switch works fine for me with systemd-boot. @Atemu which NixOS release did you experience this on?

mktemp: failed to create file via template ‘/mnt/tmp.p7wN2peYyL/tmp.XXXXXXXXXX’: No such file or directory

That's a different issue which is now fixed, see #73404 (comment).

Atemu · 2023-01-02T07:53:07Z

@ncfavier my issue was that you need NIXOS_INSTALL_BOOTLOADER which I wasn't aware of.

ncfavier · 2023-01-02T10:11:27Z

You can actually do nixos-rebuild switch --install-bootloader which is slightly better UX

dminuoso · 2023-02-15T18:32:06Z

I'm a bit unsure, this currently breaks our deployment flow.

We generate a configuration nix that contains this:

{ ... }:
{
    imports = [ ./hardware-configuration.nix ];
    virtualisation.vmware.guest.enable = true;
    virtualisation.vmware.guest.headless = true;

    boot.loader.systemd-boot.enable = true;
    boot.loader.efi.canTouchEfiVariables = true;

    fileSystems."/boot".device = "/dev/disk/by-label/nixcom-boot";
    fileSystems."/boot".fsType = "vfat";
    fileSystems."/".device = "/dev/disk/by-label/nixcom-system";
    fileSystems."/".fsType = "ext4";

    networking.useDHCP = false;
    networking.interfaces.ens32.ipv4.addresses = [ {
        address = "1.2.3.4";
        prefixLength = 27;
    } ];
    networking.defaultGateway = "1.2.3.4";
    networking.nameservers = ["1.1.1.1" "9.9.9.9"];

    services.openssh.enable = true;

    swapDevices =
        [ { device = "/dev/disk/by-label/nixcom-swap"; }
        ];

    security.sudo.wheelNeedsPassword = false;
    nix.trustedUsers = [ "@wheel" ];

    users.mutableUsers = false;

    # Essentially disable root password. Nothing can hash to "!".
    users.users.root.hashedPassword = "!";
    
    users.users."dminuoso".isNormalUser = true;
    users.users."dminuoso".extraGroups = [ "wheel" ];
    users.users."dminuoso".openssh.authorizedKeys.keys = [ ... ];

            
    system.stateVersion = "22.11";
}

Then we execute a nixos-install --no-root-password, but any subsequent colmena takeover fails with the same errors. The original installer image uses nixos-22.11, and the colmena closure as well.

dminuoso · 2023-11-24T22:45:49Z

@colemickens

Hi, how is this completed? Was it fixed by some commit on master?

colemickens · 2023-11-24T22:48:14Z

Sorry, I feel like there were a random smattering of issues that covered it. I haven't seen this in a while, I figured it was fixed. Are folks still seeing it on new installs?

colemickens added the 0.kind: bug Something is broken label Jan 22, 2022

veprbl added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Jan 22, 2022

YellowOnion mentioned this issue Feb 13, 2022

Write tests for nixos-install. #159773

Closed

rnhmjoj mentioned this issue Sep 2, 2022

stage-1.nix / mktemp: failed to create directory via template ‘initrd-secrets.XXXXXXXXXX’: Read-only file system #73404

Closed

colemickens closed this as completed Nov 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd-boot not installing on unstable #156121

systemd-boot not installing on unstable #156121

colemickens commented Jan 22, 2022

colemickens commented Jan 22, 2022 •

edited

Loading

colemickens commented Jan 22, 2022

colemickens commented Jan 22, 2022

totoroot commented Feb 2, 2022

jonringer commented Feb 2, 2022 •

edited

Loading

YellowOnion commented Feb 13, 2022 •

edited

Loading

Atemu commented Nov 17, 2022

Atemu commented Nov 17, 2022

ncfavier commented Jan 1, 2023 •

edited

Loading

Atemu commented Jan 2, 2023

ncfavier commented Jan 2, 2023

dminuoso commented Feb 15, 2023 •

edited

Loading

dminuoso commented Nov 24, 2023

colemickens commented Nov 24, 2023

systemd-boot not installing on unstable #156121

systemd-boot not installing on unstable #156121

Comments

colemickens commented Jan 22, 2022

Describe the bug

Steps To Reproduce

Expected behavior

colemickens commented Jan 22, 2022 • edited Loading

colemickens commented Jan 22, 2022

colemickens commented Jan 22, 2022

totoroot commented Feb 2, 2022

jonringer commented Feb 2, 2022 • edited Loading

YellowOnion commented Feb 13, 2022 • edited Loading

Atemu commented Nov 17, 2022

Atemu commented Nov 17, 2022

ncfavier commented Jan 1, 2023 • edited Loading

Atemu commented Jan 2, 2023

ncfavier commented Jan 2, 2023

dminuoso commented Feb 15, 2023 • edited Loading

dminuoso commented Nov 24, 2023

colemickens commented Nov 24, 2023

colemickens commented Jan 22, 2022 •

edited

Loading

jonringer commented Feb 2, 2022 •

edited

Loading

YellowOnion commented Feb 13, 2022 •

edited

Loading

ncfavier commented Jan 1, 2023 •

edited

Loading

dminuoso commented Feb 15, 2023 •

edited

Loading