boot.initrd.secrets omits key file and fails during initial installation

[ghost]

Describe the bug

Unfortunately I can not pinpoint the issue, but it seems that after a nixos-rebuild switch --upgrade on Feb 1st, boot.initrd.secrets starts to omit a key file out of 3 key files and completely fails with a fresh system installation.

Steps To Reproduce

Steps to reproduce the behavior:

1. configuration.nix contains the following lines

  boot.initrd.secrets = {
    "/etc/cryptkey.d/rpool_${poolUuid}-nixos-key-zfs" = null;
    "/etc/cryptkey.d/bpool_${poolUuid}-key-luks" = null;
    "/etc/cryptkey.d/${diskName}-part4-key-luks-swap" = null;
  };

All three files exist on disk.

2. nixos-rebuild switch produces a initrd-secret archive that is missing the third key file.
3. Fresh system installation produces the following:

+ nixos-enter --root /mnt -- /run/current-system/bin/switch-to-configuration boot
setting up /etc...
updating GRUB 2 menu...
mktemp: failed to create directory via template ‘/mnt/tmp.torvrvFbQ5/initrd-secrets.XXXXXXXXXX’: No such file or directory
failed to create initrd secrets: No such file or directory
+ rm -rf /mnt/tmp.torvrvFbQ5

Expected behavior

All key files are included in generated initrd-secrets and nixos-install --root=/mnt does not fail.

Additional context

This issue does not exist before system update.

Notify maintainers

This option is declared in <nixpkgs/nixos/modules/system/boot/stage-1.nix>.

@dasJ @grahamc

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.17, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11.335665.0f316e4d72d"`
 - channels(user0): `"home-manager-21.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[ghost]

I specifically installed an encrypted root on zfs system with this article. GRUB will decrypt disk on boot and initrd will re-decrypt disk partitions and swap with the keys embedded here.

[maxbrunet]

I have faced that too, there seem to be a problem with TMPDIR here:

nixpkgs/nixos/modules/system/boot/stage-1.nix

Line 432 in ac8ded5

tmp=$(mktemp -d ''${TMPDIR:-/tmp}/initrd-secrets.XXXXXXXXXX)

Not sure where it is set from, but its value should be relative to the chroot, e.g. /tmp.torvrvFbQ5/initrd-secrets.XXXXXXXXXX, not /mnt/tmp.torvrvFbQ5/initrd-secrets.XXXXXXXXXX.

It can be worked around by setting TMPDIR to an existing directory before installing/rebuilding, for example:

export TMPDIR=/tmp

Or by preparing a directory with mktemp yourself:

tmp="$(mktemp -d /mnt/tmp.XXXXXXXXXX)"
export TMPDIR="${tmp#/mnt}"
echo "${TMPDIR}" # /tmp.torvrvFbQ5

[maxbrunet]

May be a duplicate of #73404

[ncfavier]

Duplicate of #73404 (comment) indeed.