Unstable NAR hash for git inputs using export-subst

[mcwitt]

Describe the bug

When Flake inputs are git repos that use the export-subst feature in .gitattributes (for example, projects that use python-versioneer), the NAR hashes of these inputs can be unstable over time, leading to NAR hash mismatch in input errors.

Steps To Reproduce

Consider the flake

{
  inputs.export-subst-minimal.url = "git+https://github.com/mcwitt/export-subst-minimal";
  inputs.export-subst-minimal.flake = false;
  outputs = _: { };
}

(the input mcwitt/export-subst-minimal is a minimal example of a repo using export-subst)

The contents of the resulting store path are not reproducible:

$ nix flake lock
$ nix repl
…
nix-repl> :lf .
Added 8 variables.

nix-repl> inputs.export-subst-minimal.outPath
"/nix/store/iy7my74arsc4rgk9xrcs9mxrb0460x8d-source"

$ cat /nix/store/iy7my74arsc4rgk9xrcs9mxrb0460x8d-source/nondeterministic
 (HEAD -> main)

The contents will change for example when the pinned commit is no longer HEAD in main, leading to hash mismatch next time we fetch the input (on a different machine, or if the local cache is invalidated).

Expected behavior

Inputs using the git+https scheme should have stable NAR hashes. In particular, git-archive / export-subst actions should not be run, since these can lead to non-reproducibility as in the example above.

nix-env --version output

nix-env (Nix) 2.11.1

Additional context

• Original question: https://discourse.nixos.org/t/how-to-deal-with-unstable-flake-input-hashes-due-to-export-subst/24623
• Example workaround in nixpkgs (does not appear to work for Flake inputs): https://github.com/NixOS/nixpkgs/blob/971e45b76ab31f43891ca2ad5fb48fbd8578f654/pkgs/development/python-modules/limits/default.nix#L29-L34

Priorities

Add 👍 to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-deal-with-unstable-flake-input-hashes-due-to-export-subst/24623/3

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-01-13-nix-team-meeting-minutes-23/24644/3

[roberth]

A different problem with unreproducible git fetchTree was discussed in today's Nix Team meeting.

Another, somewhat-related data point for the github: type is that tree archives appear to receive less or no special processing.
Example: https://github.com/mcwitt/export-subst-minimal/archive/4e289bdc5e251b8aea07ba6cbc6b892450d95600.tar.gz

The tree hash could be stored in the lock file and it would also make subdir fetching more efficient.

[roberth]

Detailed notes: #4635 (comment)

[mcwitt]

Another, somewhat-related data point for the github: type is that tree archives appear to receive less or no special processing.

I think github: type inputs are still susceptible to a similar issue, though; in this case the substitution is done by github when generating the tree archive. The archives are apparently generated on demand and cached for a short period, so it's still possible to hit the NAR hash mismatch in input issue after some time. (Actually, I initially found this using github: and attempted a workaround by switching to git+https:, hoping the latter would be reproducible)

EDIT: sorry, I'd misunderstood! Indeed, it looks like https://github.com/mcwitt/export-subst-minimal/archive/4e289bdc5e251b8aea07ba6cbc6b892450d95600.tar.gz does not have the substitution applied

[Mic92]

If we were using libgit2 would this give us more control over this?

[roberth]

It does, and we've been using libgit2 since Nix 2.20

• Introduce libgit2 #9240

nix-repl> builtins.readFile (builtins.fetchTree { type = "git"; url = "https://github.com/mcwitt/export-subst-minimal"; } + "/nondeterministic")
"$Format:%d$\n"

This is what flakes does internally as well.

Solved since #9240