fetchFromGitHub output changed -- cannot build lua-mode

[vlaci]

Issue description

I cannot install emacsPackagesNg.melpaPackages.lua-mode anymore. I am pretty sure it worked on an another machine a couple of weeks ago. Interestingly the src definition was last modified over 10 months ago in b801736. The revision before that fails to build too so the issue seems to be tied to the upstream repository.

It seams to me that the archive downloaded by fetchFromGitHub itself has changed. I don't know if such breakages are meant to happen but it worries me a bit. If it is something that worth more investigation I may be able to pull the archive having the correct checksum from an another machine to check how the newer one differs from it.

Steps to reproduce

The following command fails to run on a fresh (d359b5d) nixpkgs local checkout and on release-18.09:

nix-build ./. -A emacsPackagesNg.melpaPackages.lua-mode
these derivations will be built:
  /nix/store/bhq7kby9lyyv9vgdf1683kw2ckn6akgw-source.drv
  /nix/store/v4il31qzhry1spkj3p5kh80igd4w8fz8-recipe.drv
  /nix/store/p51ydqzrlv3wv57q1vcvd93y36hb8hi0-emacs-lua-mode-20180323.321.drv
building '/nix/store/v4il31qzhry1spkj3p5kh80igd4w8fz8-recipe.drv'...
building '/nix/store/bhq7kby9lyyv9vgdf1683kw2ckn6akgw-source.drv'...

trying https://github.com/immerrr/lua-mode/archive/99312b8d6c500ba3067da6d81efcfbbea05a1cbd.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

trying https://raw.githubusercontent.com/milkypostman/melpa/ca7bf43ef8893bf04e9658390e306ef69e80a156/recipes/lua-mode
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   158    0   158    0     0    511      0 --:--:-- --:--:-- --:--:--   511
100    54  100    54    0     0    134      0 --:--:-- --:--:-- --:--:--   134
100 35311    0 35311    0     0  37013      0 --:--:-- --:--:-- --:--:-- 21.0M
unpacking source archive /build/99312b8d6c500ba3067da6d81efcfbbea05a1cbd.tar.gz
fixed-output derivation produced path '/nix/store/7kw98xzwm0l6dqqmk32x8xpyc82f03y1-source' with sha256 hash '1gi8k2yydwm1knq4pgmn6dp92g97av4ncb6acrnz07iba2r34dyn' instead of the expected hash '04m9njcpdmar3njjz4x2qq26xk0k6qprcfzx8whlmvapqf8w19iz'
cannot build derivation '/nix/store/p51ydqzrlv3wv57q1vcvd93y36hb8hi0-emacs-lua-mode-20180323.321.drv': 1 dependencies couldn't be built
error: build of '/nix/store/p51ydqzrlv3wv57q1vcvd93y36hb8hi0-emacs-lua-mode-20180323.321.drv' failed

Technical details

• system: "x86_64-linux"
• host os: Linux 4.19.17, NixOS, 18.09.1985.749a3a0d00b (Jellyfish)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.1.3
• channels(root): "nixos-unstable-19.03pre166987.bc41317e243, nixos-18.09.1922.97e0d53d669, blox, home-manager-18.09, mozilla, nixos-hardware"
• channels(vlaci): "mozilla, nixos-unstable-19.03pre162671.44b02b52ea6"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[infinisil]

That's really weird. One might think that emacs2nix could be at fault here, so I tried to check this by verifying the hashes of all emacs packages, hoping to find a couple more where this would happen. So here's the results:

On bf041c3, out of the 4236 attributes in emacsPackagesNg where src is not null:

• 7 of them have a hash mismatch:
  • lua-mode, url: https://github.com/immerrr/lua-mode/archive/99312b8d6c500ba3067da6d81efcfbbea05a1cbd.tar.gz
  • filesets-plus, url: https://www.emacswiki.org/emacs/download/filesets%2b.el
  • header2, url: https://www.emacswiki.org/emacs/download/header2.el
  • hexrgb, url: https://www.emacswiki.org/emacs/download/hexrgb.el
  • lib-requires, url: https://www.emacswiki.org/emacs/download/lib-requires.el
  • thingatpt-plus, url: https://www.emacswiki.org/emacs/download/thingatpt+.el
  • yaoddmuse, url: http://emacswiki.org/emacs/download/yaoddmuse.el
• 2 of them give a 404:
  • tern-django
  • flycheck-purescript

The emacswiki ones make sense, they don't seem to use persistent urls. But out of the almost 4000 github sources for these packages, only lua-mode has a hash mismatch! Even though we explicitly requested a specific git hash.

So, either

• emacs2nix has a bug (or nix-prefetch-url, which is what it uses to get the hashes for GitHub), but considering above, this is unlikely
• GitHub has a bug, in that it either
  • served the wrong git version at some point (even though we explicitly requested a specific revision, which should give us the same hash everytime)
  • served a tarball for the same git revision that had some files different (maybe just an execute bit)
• Nix' tarball unpacker has a bug, in that it doesn't properly normalize all files in the tarball (GitHub server tarballs with different hashes, but the unpacked files should always be the same)

All of those are very unlikely unfortunately..

I think the best thing to do for now is:

• Do a quick check of lua-mode's source code before fixing the hash, just to be sure
• Open an issue in lua-mode, asking if they noticed anything suspicious recently
• If that doesn't reveal anything, ask GitHub support for whether they have some more info on this

[oxij]

If you had it installed before and still have the source in /nix/store you can also just diff it with the new source. Btw, the rejected #49862 simplifies such things a lot (if you keep it applied for a while).

[vlaci]

Unfortunately the older sources got garbage collected so I cannot check what is changed in them.

[terlar]

I can confirm that I experience the same thing.

I tried diffing the two sources:

diff --brief -Nr /nix/store/7kw98xzwm0l6dqqmk32x8xpyc82f03y1-source /nix/store/sc08qxggk9nwj730s20yz8q81d7gfjgy-source

git diff --no-index /nix/store/7kw98xzwm0l6dqqmk32x8xpyc82f03y1-source /nix/store/sc08qxggk9nwj730s20yz8q81d7gfjgy-source

But it doesn't show any difference. However, I am not 100% sure if the diff command is doing the correct thing

[terlar]

Trying to debug this I checked this, which I guess is one of the steps which I guess is what @infinisil was doing:

$ nix-prefetch-git --no-deepClone git://github.com/immerrr/lua-mode.git 95c64bb5634035630e8c59d10d4a1d1003265743
Initialized empty Git repository in /tmp/git-checkout-tmp-ONpIaN9L/lua-mode-95c64bb/.git/
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 24 (delta 0), reused 10 (delta 0), pack-reused 0
Unpacking objects: 100% (24/24), done.
From git://github.com/immerrr/lua-mode
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...

git revision is 95c64bb5634035630e8c59d10d4a1d1003265743
path is /nix/store/k8q2802zz5y6x4zk6nkxcdfybgpj7qs5-lua-mode-95c64bb
git human-readable version is -- none --
Commit date is 2019-01-13 13:50:39 +0300
hash is 1mra4db25ds64526dsj8m5yv0kfq3lgggjh1x6xmqypdaryddbcp
{
  "url": "git://github.com/immerrr/lua-mode.git",
  "rev": "95c64bb5634035630e8c59d10d4a1d1003265743",
  "date": "2019-01-13T13:50:39+03:00",
  "sha256": "1mra4db25ds64526dsj8m5yv0kfq3lgggjh1x6xmqypdaryddbcp",
  "fetchSubmodules": false
}

When I searched for the new hash on github I found @jwiegley had added the new hash as an override:
jwiegley/nix-config@103bcb6#diff-1fbf255f21bddf22c0f9c81fbe448bcfR823

So I am doing the same for now. In case anyone wants a temporary workaround.

[endgame]

Any progress on this? While I can at least hack around the expected hash for lua-mode, it's a bit sad to have a bug like this as we head into 19.03.