nix-store --add-fixed no longer accepts symlinks

[jdelStrother]

This was originally filed as a macos-specific bug ("nix-prefetch-git symlink error on macos"), but I think the root cause is a universal issue.

Describe the bug

As of nix 2.20, you can't call nix-store --add-fixed on a path with a symlink parent.

Steps To Reproduce

~/dev $ nix-shell -p 'nixVersions.nix_2_20'

[nix-shell:~/dev]$ mkdir -p foo/bar && ln -s foo foo-link

[nix-shell:~/dev]$ nix-store --add-fixed --recursive sha256 foo-link/bar
error: path '«unknown»/Users/jon/dev/foo-link' is a symlink

The same command works fine with 2.19

This causes problems with tools like nix-prefetch-git, especially on macOS where it clones the repo into /tmp, which is actually a symlink to /private/tmp.

The change-in-behaviour presumably originates from 83c067c

Original issue details

Describe the bug

nix-prefetch-git doesn't seem to work for me on macos with the 24.11-beta tag - it complains that /tmp is a symlink (which it is, it links to /private/tmp).

Steps To Reproduce

$ nix run github:nixos/nixpkgs/24.11-beta#nix-prefetch-git -- --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256
Initialized empty Git repository in /private/tmp/git-checkout-tmp-MzN6ZcNg/polo-316aa2a/.git/
remote: Enumerating objects: 51, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 51 (delta 8), reused 19 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (51/51), 19.57 KiB | 668.00 KiB/s, done.
From https://github.com/IFTTT/polo
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...
error: path '/tmp' is a symlink

Expected behavior

Successfully fetches a git repo into the store

Additional context

The same command works fine with 24.05:

$ nix run github:nixos/nixpkgs/24.05#nix-prefetch-git -- --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256
Initialized empty Git repository in /private/tmp/git-checkout-tmp-22f9iAdj/polo-316aa2a/.git/
remote: Enumerating objects: 51, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 51 (delta 8), reused 19 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (51/51), 19.57 KiB | 715.00 KiB/s, done.
From https://github.com/IFTTT/polo
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...

git revision is 316aa2ac210a45a7fc400ab921831493d5dd21b8
path is /nix/store/nljjjs2idzm9j5iy9swppd37z6aq96fb-polo-316aa2a
git human-readable version is -- none --
Commit date is 2022-04-04 16:29:34 -0700
hash is 1vikzclg64lpydfwvz2imzwvl2f2vv1iq9ww43ah31p84w2xfmih
{
  "url": "https://github.com/IFTTT/polo.git",
  "rev": "316aa2ac210a45a7fc400ab921831493d5dd21b8",
  "date": "2022-04-04T16:29:34-07:00",
  "path": "/nix/store/nljjjs2idzm9j5iy9swppd37z6aq96fb-polo-316aa2a",
  "sha256": "1vikzclg64lpydfwvz2imzwvl2f2vv1iq9ww43ah31p84w2xfmih",
  "hash": "sha256-MFbXBSfohgHVIJwnHMPewgm6+a9R/M1d85cS8yj7M+4=",
  "fetchLFS": false,
  "fetchSubmodules": false,
  "deepClone": false,
  "leaveDotGit": false
}

I'd originally posted this in nixpkgs, but then bisected it down to 9a94e07, so I'm guessing the problem is actually in nix itself..?

Metadata

nix-env (Nix) 2.18.8

(though I believe that nix-prefetch-git is using 2.24, at least in the 22.11 release

Checklist

• checked [latest Nix manual] ([source])
• checked [open bug issues and pull requests] for possible duplicates

Add 👍 to issues you find important.

[jdelStrother]

Assuming this is an intentional behaviour change, I've submitted NixOS/nixpkgs#358685 so that the nix-prefetch scripts resolve /tmp to a real directory before calling nix-store --add-fixed on it

[jdelStrother]

Speaking more generally, is it expected that you can't call nix-store --add-fixed on a path with a symlink parent as-of nix.2.20, on nixos as well as macos ?

~/dev $ nix-shell -p 'nixVersions.nix_2_20'

[nix-shell:~/dev]$ mkdir -p foo/bar && ln -s foo foo-link

[nix-shell:~/dev]$ nix-store --add-fixed --recursive sha256 foo-link/bar
error: path '«unknown»/Users/jon/dev/foo-link' is a symlink

I don't see a relevant-sounding change listed in the release notes, not sure if it's a bug or feature.

[NobbZ]

The last comment gives a way to reproduce on linux easily.

• 83c067c
• 345f79d
• 504e4fc
• 57246c4

The last 2 even mentioning symlinks in their commit messages

Bisection log

# bad: [16e1ff3bcb5e785cce6bff8d0745e2a73fbd99de] Mark as stable
# good: [5b99c823ef95ba5c642ae105815d5acd4f093aa3] Mark official release
git bisect start '2.20.0' '2.19.0'
# good: [6832d18ac734f4b855f97c07b158491dd01cefcd] Merge pull request #9399 from edolstra/revert-vlas
git bisect good 6832d18ac734f4b855f97c07b158491dd01cefcd
# bad: [16a77767cdf90cbd53ff451c93fde8e5d89bdb1f] Merge pull request #9635 from 9999years/document-makefile-variables
git bisect bad 16a77767cdf90cbd53ff451c93fde8e5d89bdb1f
# good: [9817788f6b25fefa731b838391f859e886ae0d24] Merge pull request #9518 from NixOS/fix-static-build
git bisect good 9817788f6b25fefa731b838391f859e886ae0d24
# bad: [b1c633c6bb2e2dbc8a2891edf0d0c6f9d31d890b] Merge pull request #9600 from SharzyL/fix_nix_copy
git bisect bad b1c633c6bb2e2dbc8a2891edf0d0c6f9d31d890b
# bad: [7cdc8786d9f57410acbec088e13e46ce10cdce35] Merge pull request #9568 from hercules-ci/revert-9553
git bisect bad 7cdc8786d9f57410acbec088e13e46ce10cdce35
# good: [139982997eec493a0f74105c427953f6be77da6d] Merge pull request #9560 from obsidiansystems/serve-proto-unkeyed-valid-path-info-serializer
git bisect good 139982997eec493a0f74105c427953f6be77da6d
# bad: [2bd83225004012af97d2d5977dc1de952f60aa8d] Update src/libfetchers/filtering-input-accessor.hh
git bisect bad 2bd83225004012af97d2d5977dc1de952f60aa8d
# skip: [345f79d01676680f2d4ef8803790896a190c855b] Check that we can't follow symlinks outside of the allowed paths
git bisect skip 345f79d01676680f2d4ef8803790896a190c855b
# skip: [504e4fc4576dc6a4cd5c083a3bf7b80dfb0ca220] CanonPath: Support std::hash
git bisect skip 504e4fc4576dc6a4cd5c083a3bf7b80dfb0ca220
# skip: [83c067c0fa0cc5a2dca440e5c986afe40b163802] PosixSourceAccessor: Don't follow any symlinks
git bisect skip 83c067c0fa0cc5a2dca440e5c986afe40b163802
# bad: [57246c4c3802920e6167fd540dae2a0abca97f15] PosixSourceAccessor: Cache lstat() calls
git bisect bad 57246c4c3802920e6167fd540dae2a0abca97f15
# good: [8cafc754d845529a78595d1196769257ee23ca56] Move access control from FSInputAccessor to FilteringInputAccessor
git bisect good 8cafc754d845529a78595d1196769257ee23ca56
# only skipped commits left to test
# possible first bad commit: [57246c4c3802920e6167fd540dae2a0abca97f15] PosixSourceAccessor: Cache lstat() calls
# possible first bad commit: [504e4fc4576dc6a4cd5c083a3bf7b80dfb0ca220] CanonPath: Support std::hash
# possible first bad commit: [83c067c0fa0cc5a2dca440e5c986afe40b163802] PosixSourceAccessor: Don't follow any symlinks
# possible first bad commit: [345f79d01676680f2d4ef8803790896a190c855b] Check that we can't follow symlinks outside of the allowed paths

[ofalvai]

I believe this is the same issue as #11681

[9999years]

Full list of duplicates (that I can find):

• nix-prefetch-git seems broken on macos in release-24.11 nixpkgs#358113
• Hashing values not in a regular file/directory #11756
• nix-hash process substitution example fails with error: path '/dev/fd/63' is a symlink on Arch Linux #11681

Possibly related:

• Symlinks to flakes have poor UX #9253
• bug: nix commands do not follow flake-url symlinks #9463
• nix-shell regression for shell.nix symlinks to other directories #11630

This bug appears to have a wide blast radius. This is the exact reason there was so much resistance to moving the Nixpkgs default Nix version away from 2.18 and makes the decision to remove versions of Nix older than 2.24 from Nixpkgs (despite known regressions with no workarounds) so much more frustrating.