fetchTree has inconsistent behavior between Linux and Darwin (libarchive behavior)

[bryanhonof]

cc: @tomberek

Describe the bug

When Nix tries to fetch a git repository that contains a so called AppleDouble file, it'll filter out that file on Darwin machines, but it will copy them over on Linux machines. This results in a NAR hash mismatch.

Steps To Reproduce

1. Create a git repository that contains a file—with some contents in it—that has a ._ (dot underscore) as a prefix
2. Use builtins.fetchTree to fetch said repo on a Linux machine
3. Observe the NAR hash
4. Repeat steps 2 and 3 on a Linux machine
5. Notice the NAR hashes are different, whilst everything else stays the same

$ uname
Darwin
$ nix --refresh eval --impure --expr 'builtins.fetchTree {type="github";owner="bryanhonof";repo="AppleDoubleNix";}'
{ lastModified = 1699021234; lastModifiedDate = "20231103142034"; narHash = "sha256-oeP6/ZIGPYSA/hYmaLRODYF2eWF8/PcOJH9QTVwXHoY="; outPath = "/nix/store/xqcn1zm02sl84z3f7ay6dixjyig449qg-source"; rev = "5acb60fa3290ff4eb6c3f62706cb71d06b0e4831"; shortRev = "5acb60f"; }

$ uname
Linux
$ nix --refresh eval --impure --expr 'builtins.fetchTree {type="github";owner="bryanhonof";repo="AppleDoubleNix";}'
{ lastModified = 1699021234; lastModifiedDate = "20231103142034"; narHash = "sha256-uqXwlm5CG+qFI56WG3ezrVJ8i9zlQSvZ5bn+oEM/1w0="; outPath = "/nix/store/qr25ihyhcvqwqlh8079p51yscgwrbs1w-source"; rev = "5acb60fa3290ff4eb6c3f62706cb71d06b0e4831"; shortRev = "5acb60f"; }

Expected behavior

The NAR hashes for source code should be the same, regardless of architecture.

nix-env --version output

nix-env (Nix) 2.15.1

Additional context

• AppleDouble files are files that start with a ._
• libarchive has logic to ignore AppleDouble files
• Nix uses libarchive
• libarchive upstream issue question: Would it be possible to bypass AppleDouble behavior? libarchive/libarchive#2010

Priorities

Add 👍 to issues you find important.

[tomberek]

Possible fix:

diff --git a/src/libutil/tarfile.cc b/src/libutil/tarfile.cc
index 1733c791c..1ce42221f 100644
--- a/src/libutil/tarfile.cc
+++ b/src/libutil/tarfile.cc
@@ -53,6 +53,7 @@ TarArchive::TarArchive(Source & source, bool raw) : buffer(65536)
         archive_read_support_format_raw(archive);
         archive_read_support_format_empty(archive);
     }
+    archive_read_set_format_option(this->archive, NULL ,"mac-ext",NULL);
     check(archive_read_open(archive, (void *)this, callback_open, callback_read, callback_close), "Failed to open archive (%s)");
 }
 
@@ -63,6 +64,7 @@ TarArchive::TarArchive(const Path & path)
 
     archive_read_support_filter_all(archive);
     archive_read_support_format_all(archive);
+    archive_read_set_format_option(this->archive,NULL, "mac-ext",NULL);
     check(archive_read_open_filename(archive, path.c_str(), 16384), "failed to open archive: %s");
 }
 

[tomberek]

Related: #9327 (comment)