stage-1.nix / mktemp: failed to create directory via template ‘initrd-secrets.XXXXXXXXXX’: Read-only file system

[h0m1]

When "nixos-rebuild boot" is executed from /nix/store (cwd is readonly) it fails with an error message.

It works when you call it from a writeable directory.

In my case the problem only occurs, when the option "boot.initrd.network.ssh" is enabled. Without it works.

What also happend is a broken bootloader. The system did not boot anymore after this failure. A reinstallation of the bootloader was needed.

To Reproduce

1. activate the following options:

boot.initrd.network.enable = true;
boot.initrd.network.ssh.enable = true;
boot.initrd.network.ssh.port = 5022;
boot.initrd.network.ssh.hostECDSAKey = /etc/nixos/secrets/ecdsa-hostkey;

2. open a shell and change the working directory to /nix/store

3. execute "nixos-rebuild boot"

Expected behavior
nixos-rebuild should work from every working directory.

Additional context
I think it is related to function "initialRamdiskSecretAppender" in "nixos/modules/system/boot/stage-1.nix"

The mktemp-call creates a directory in the working directory.

It only happens when config.boot.initrd.secrets is not empty.

Metadata

nix run nixpkgs.nix-info -c nix-info -m

• system: "x86_64-linux"
• host os: Linux 4.19.81, NixOS, 20.03pre200231.7827d3f4497 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.1
• channels(micha): ""
• channels(root): "nixos-20.03pre200231.7827d3f4497"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

console.txt

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[antifuchs]

I'm getting bitten by this as well, trying to install initrd secrets via a remote-deploy method that doesn't really let me control CWD.

[stigok]

Bit me today. Great tips that this only happens when executing e.g. sudo nixos-rebuild switch from a non-writable directory, like cwd being /nix/store/...... Worked well again when running from $HOME.

[colemickens]

I'm confused. I'm in /tmp, it's write-able, and I'm hitting this?

[nixos@nixos:/tmp]$ sudo nixos-install --system /nix/store/6y25ndxn9w2biih9g0yrdk77ald72gdg-nixos-system-sinkor-21.11.20210829.784b978 --root /mnt --no-root-passwd
copying channel...
installing the boot loader...
setting up /etc...
setting up secrets...
/nix/store/58d0sjjy6gdq1ryfgrsqsjym7b83zbhc-sops-install-secrets-0.0.1/bin/sops-install-secrets: Error setting up gpg keyring: Cannot read ssh key '/etc/ssh/ssh_host_rsa_key': open /etc/ssh/ssh_host_rsa_key: no such file or directory
Activation script snippet 'setup-secrets' failed (1)
Copied "/nix/store/9xnv1q1v89bkx4g65dfasknzsxglxpp2-systemd-247.6/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/systemd/systemd-bootaa64.efi".
Copied "/nix/store/9xnv1q1v89bkx4g65dfasknzsxglxpp2-systemd-247.6/lib/systemd/boot/efi/systemd-bootaa64.efi" to "/boot/EFI/BOOT/BOOTAA64.EFI".
Random seed file /boot/loader/random-seed successfully written (512 bytes).
mktemp: failed to create directory via template ‘/mnt/tmp.kYftcsVwDN/initrd-secrets.XXXXXXXXXX’: No such file or directory
Traceback (most recent call last):
  File "/nix/store/ab23bcyfrk9d82lz8b9gl2ljhm4hv5ki-systemd-boot", line 274, in <module>
    main()
  File "/nix/store/ab23bcyfrk9d82lz8b9gl2ljhm4hv5ki-systemd-boot", line 240, in main
    write_entry(*gen, machine_id)
  File "/nix/store/ab23bcyfrk9d82lz8b9gl2ljhm4hv5ki-systemd-boot", line 103, in write_entry
    subprocess.check_call([append_initrd_secrets, "/boot%s" % (initrd)])
  File "/nix/store/mj50n3hsqrgfxjmywsz4ymhayjfpqlhf-python3-3.9.6/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/dnhvaarm5h7ckc2x54c6pkifsqasrp4l-append-initrd-secrets/bin/append-initrd-secrets', '/boot/efi/nixos/ml7c0qa9zwshlzv79k56jm3d5lh544g4-initrd-linux-5.13.13-initrd.efi']' returned non-zero exit status 1.

[0x4A6F]

Strange, I'm getting this behavior:

nixos-install -I nixpkgs=/mnt/root/nixpkgs # git checkout 8d8a28b

building the configuration in /mnt/etc/nixos/configuration.nix...
these derivations will be built:
  /nix/store/rf22095jyf7qbri0w56msw1z915hag3v-append-initrd-secrets.drv
  /nix/store/211h9sqlisr18v8l7f9w1jxq0hkl60lj-nixos-system-instance-aarch64-nixos-21.11pre-git.drv
building '/nix/store/rf22095jyf7qbri0w56msw1z915hag3v-append-initrd-secrets.drv'...
building '/nix/store/211h9sqlisr18v8l7f9w1jxq0hkl60lj-nixos-system-instance-aarch64-nixos-21.11pre-git.drv'...
/nix/store/f18wv96f7vrkv0md5mf1zq3qwrgzh7an-nixos-system-instance-aarch64-nixos-21.11pre-git
copying channel...
installing the boot loader...
setting up /etc...
/etc/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
updating GRUB 2 menu...
mktemp: failed to create directory via template ‘/mnt/tmp.DiGCku8IJk/initrd-secrets.XXXXXXXXXX’: No such file or directory
failed to create initrd secrets: No such file or directory

With reverted #73780 the installation works (i.e. changing following in /mnt/root/nixpkgs/nixos/modules/system/boot/stage-1.nix and clearing /mnt/nix).

- tmp=$(mktemp -d ''${TMPDIR:-/tmp}/initrd-secrets.XXXXXXXXXX)
+ tmp=$(mktemp -d initrd-secrets.XXXXXXXXXX)

nixos-install -I nixpkgs=/mnt/root/nixpkgs

installing the GRUB 2 EFI boot loader into /boot...
Installing for arm64-efi platform.
/nix/store/0k0zwn7imdnvfm9i4v3k4nfrynjhm45s-grub-2.06/sbin/grub-install: warning: cannot open directory `/nix/store/0k0zwn7imdnvfm9i4v3k4nfry
njhm45s-grub-2.06/share/locale': No such file or directory.
Installation finished. No error reported.
setting up /etc...
/etc/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
setting up /etc...
/etc/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
setting root password...
New password:
Retype new password:
passwd: password updated successfully
installation finished!

[colemickens]

I'm on a fresh install and once again, nixos-install from the top doesn't work, and I have to sudo nixos-enter and then execute the nixos-install (when inside the chroot, use --root /), and then it installs successfully...

[tomberek]

Perhaps re-use this:

nixpkgs/nixos/modules/system/boot/stage-1.nix

Line 378 in 4df4429

tmp=$(mktemp -d ''${TMPDIR:-/tmp}/initrd-secrets.XXXXXXXXXX)

tmp=$(mktemp -d ''${TMPDIR:-/tmp}/initrd-secrets.XXXXXXXXXX)

Or rather we need some more info about what the exact failure is. Does /tmp exist? How is it mounted? TMPDIR? etc.

This error (mktemp: failed to create directory via template ‘/mnt/tmp.DiGCku8IJk/initrd-secrets.XXXXXXXXXX’: No such file or directory) suggests the tmp is built in the mount space during nixos-install.

[snkgak]

I'm hitting this same problem using initrd.secrets, trying to install 21.11. Same config works fine on 21.05. Happens right after "updating GRUB 2 menu ..." Kills the GRUB 2 install ...

BTW, @colemickens method of sudo nixos-enter followed by nixos-install --root / allowed GRUB 2 to install and complete the install - Thanks!

[Shados]

I'm confused. I'm in /tmp, it's write-able, and I'm hitting this?

/tmp exists and is writable, but take a close look at this error line: mktemp: failed to create directory via template ‘/mnt/tmp.kYftcsVwDN/initrd-secrets.XXXXXXXXXX’: No such file or directory

It's attempting to create a file in a /tmp.kYftcsVwDN directory in the target root, which I am guessing does not exist, and mktemp then fails because it refuses to create parent directories. What is unclear to me is why $TMPDIR is apparently resolving to /mnt/tmp.kYftcsVwDN. How is this being set...?

[Shados]

Ah, I found it, it's set by nixos-install itself here. The temporary directory it creates does get created, but somehow ceases to exist prior to the grub installation running. Hmm... Ah, of course, the grub installation is run within a nixos-enter chroot environment, so the temporary directory path is no longer valid (because there is no /mnt/tmp.kYftcsVwDN within the chroot, only outside of it).

EDIT: I feel like this is arguably a bug in nixos-enter. Should it really be retaining external environment variables...?

[WolfangAukang]

FYI, today I tried to install NixOS following ladinu's guide and got a similar error, but instead of Read-only file system, is the same issue indicated by @Shados (No such file or directory).

I'm going to try this with a 21.05 image I have available to see if that makes any kind of difference.

UPDATE: I didn't find the 21.05 ISO, so I followed @colemickens advice. It worked, although I'm not sure about the uninitialized /etc warnings.

[jaen]

It just bit me as well, solved with a hacky sudo mount --bind /mnt /mnt/mnt, but it would be nice to have a clean solution ~~"

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-nixos-install-fails-mktemp-failed-to-create-file-via-template-run-user-0-tmp-xxxxxxxxxx-no-such-file-or-directory/17176/2

[IreneKnapp]

Just ran into this. I successfully worked around it by combing everyone's workarounds. Can I do anything to help get a fix ready? I see this has been open for quite a while.

[chasecaleb]

I can confirm this is still an issue too. I can also confirm mkdir /mnt/mnt && mount --bind /mnt /mnt/ works for... whatever reason. I would love a fix too, and unfortunately it's a little over my head/availability to figure out myself.

Here's a permalink to my NixOS configuration repo at the current time of writing for context if needed:

• https://gitlab.com/chasecaleb/nixos-config/-/tree/5126a35078ed62cfb64352410ff06cc663317f37/
• My install script with the workaround highlighted: https://gitlab.com/chasecaleb/nixos-config/-/blob/5126a35078ed62cfb64352410ff06cc663317f37/bootstrap.sh#L124-127
• My NixOS boot configuration: https://gitlab.com/chasecaleb/nixos-config/-/blob/5126a35078ed62cfb64352410ff06cc663317f37/nixos/base.nix#L7-27

[rnhmjoj]

Can we have a proper fix for 22.05? This is an absolutely terrible experience for new users.

[Adrien-Luxey]

Using 22.05, following the install guide for Kimsufi and the remote unlocking guide from the wiki.

• This directive: boot.initrd.hostKeys = [ "/etc/secrets/initrd/ssh_host_rsa_key" "/etc/secrets/initrd/ssh_host_ed25519_key" ];
• The keys are in /mnt/etc/secrets/initrd/ according to the host.

I get the following error message:

mktemp: failed to create directory via template ‘/tmp/initrd-secrets.XXXXXXXXXX’: No such file or directory
failed to create initrd secrets: No such file or directory

Both proposed hacks did not work:

• A variation of nixos-enter then nixos-install --root / fails with error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using or -I)

  nixos-enter previously said chroot: failed to run command ‘systemd-tmpfiles’: No such file or directory

  Maybe I simply don't know how to chroot, maybe it's nixos-enter's fault?

• mkdir /mnt/mnt && mount --bind /mnt /mnt/ followed by an install (without nixos-enter) yields the same error message: ‘/tmp/initrd-secrets.XXXXXXXXXX’: No such file or directory

  Not surprising, given that the path did not contain /mnt... as previously reported.

[ivankovnatsky]

I think I've went further by mkdir -p /mnt/tmp/.

[rnhmjoj]

I've linked another similar error to this issue: #156121 (comment). It was reported that the same workaround (#156121 (comment) ) works for that too. So, whatever the problem is, it's getting worse: it's not limited to initrd secrets but likely anything using the /tmp during the installation.

[samueldr]

A link to the currently working workaround should be prepended to the issue body; I wanted to do it, but it seems like it's unclear which workaround is to be used.

[RaitoBezarius]

Is this a release blocker for 22.11 ? cc @mweinelt ?

[mweinelt]

Probably not, given that it has been an open issue for years and nobody provided a fix until now.

[YellowOnion]

please make it a release blocker, I'm sick of googling this issue every half year.

[IreneKnapp]

I'm not involved in the release process so my thoughts don't count for much, but I think there's a real case to make this a release blocker, given that it makes the experience worse for first-time users.

Regardless of whether it's designated a blocker, I'd strongly encourage anyone who has the background on install internals to take a look at this, both because it's one of the ugliest things I've encountered when running NixOS systems, and because it appears to violate build hygiene, which is a hole in the theoretical underpinnings of nix.

[Mikilio]

It would be nice if anyone still having this issue could try my fix in the pull request #199425 to confirm if it solves this issue as well. It might accelerate a merge if it could solve multiple issues at once.

[rnhmjoj]

confirm if it solves this issue as well

There are probably several distinct issues being reported here.
I don't know if your patch fixed the mktemp: failed to create directory... error but it doesn't solve the cp: cannot stat /mnt/path/needed/for/initrd/secrets/ one.
I still have to bind mount /mnt/path/needed/for/initrd/secrets inside /mnt/mnt for the installation to succeeds.

[Mikilio]

but it doesn't solve the cp: cannot stat /mnt/path/needed/for/initrd/secrets/ one. I still have to bind mount /mnt/path/needed/for/initrd/secrets inside /mnt/mnt for the installation to succeeds.

Do you maybe know which script causes this error?

[rnhmjoj]

@Mikilio I assume it's coming from here but I haven't checked.

[Mikilio]

@rnhmjoj Since it doesn't fail with mkdir I assume it's got something to do with some value that appears as source' to be problematic.
It might be helpful to know the value of the attribute set config.boot.initrd.secrets and check if one of the values might rely on something like mkdir /mnt/mnt && mount --bind /mnt /mnt/.

[rnhmjoj]

some value that appears as source' to be problematic.

The problem is almost surely that the activation script is running inside the chroot environment and the paths, relative to the real root, don't exist there. The solution, assuming the paths actually exist under the new root (say, /mnt) is to strip prefix (/mnt) before doing anything with them.

It might be helpful to know the value of the attribute

It should be very easy to reproduce: just put the a file in the configuration directory, say /mnt/etc/nixos/secret.key and add:

  boot.initrd.secrets."secret.key" = ./secret.key;

[Mikilio]

I should probably debug this and check for the pwd at this point in the script. Unfortunately I'm very busy with a different project and don't see me doing this in the foreseeable future.

[ghuntley]

mkdir /mnt/tmp was the solution for me. What a shitty first time user experience when doing installation.

edit: also https://discourse.nixos.org/t/nixos-21-05-installation-failed-installing-from-an-existing-distro/13627/3

[ncfavier]

Let's try to wrap up this mess. As far as I can tell there are at least five different issues being reported in this thread:

• The initial issue (errors mentioning a "Read-only filesystem") was caused by using the CWD as a temporary directory, and was fixed by stage-1: create temporary secrets directory in /tmp and not in cwd #73780 several releases ago.
• Errors of the form mktemp: failed to create directory via template ‘/mnt/tmp.kYftcsVwDN/initrd-secrets.XXXXXXXXXX’: No such file or directory were caused by nixos-enter inheriting TMPDIR as-is without stripping the /mnt prefix. This was fixed in 22.05 by 1783cfd (#150408).
• Errors of the form mktemp: failed to create directory via template ‘/tmp/...’: No such file or directory were caused by a missing /tmp directory under the mount point (The result of (nixos-install && nixos-enter) lacks the /tmp directory #39140), which was fixed in nixos/nixos-enter: Create /tmp #80769 and added dependency to systemd #199425 (this was just merged, the workaround is sudo mkdir -p /mnt/tmp).
• Errors of the form cp: cannot stat /mnt/... are caused by using relative paths in your boot.initrd.secrets configuration, which are evaluated outside the chroot and used inside the chroot. I'm not sure how to fix this, but it needs its own issue. @rnhmjoj could you create it please? In the meantime, the workaround for this (and possibly other issues) is sudo mkdir -p /mnt/mnt && sudo mount --bind /mnt /mnt/mnt.
• https://discourse.nixos.org/t/nixos-21-05-installation-failed-installing-from-an-existing-distro/13627/3 points to a different issue (nixos-install fails with getting attributes of path /nix/store/...: No such file or directory #126141), which is now closed.

As the bulk of the issue seems to be resolved, I'm going to close this issue (but leave it pinned because it might still contain helpful workarounds).

If you encounter a problem that's not listed here, please open a new issue.

I will try to add boot.initrd.secrets to the installer test. EDIT: #208643

[rnhmjoj]

On unstable the remaining issue has been fixed: so, you can finally install NixOS with relative paths in boot.initrd.secrets.
Once booted into the system, running nixos-rebuilds will warn that the first generation refers to /mnt/... that no longer exists, but the command will succeed anyway. It's a bit unfortunate but fixing this properly would require a major redesign of this system.

[IreneKnapp]

well done all! super glad to see this wrapped up!

(I know I'm late with my congratulations but it was such a long-running issue that I figure that's fine)