Can not build stack projects that depend on hpack projects

[arianvp]

Describe the bug
We do not ship the same version of hpack as is vendored with stack upstream. This causes stack.yaml.lock files to be invalid as our stack generates different cabal ifles than the stack

To Reproduce
Steps to reproduce the behavior:

1. have a stack file that has an extra-dep on a project that doesn't have cabal file .e.g https://github.com/wireapp/wire-server/blob/develop/stack.yaml#L93

2. Have lockfile mismatch because the hpack versions generate slightly different cabal files

[arian@t490s:~/Projects/wire/wire-server]$ stack build
Cloning c6233c493b1a7c3df8099872bbc1f66c5f25d95f from https://github.com/bitemyapp/bloodhound
Mismatched package metadata for Repo from https://github.com/bitemyapp/bloodhound, commit c6233c493b1a7c3df8099872bbc1f66c5f25d95f
Found: bloodhound-0.17.0.0 with cabal file 853812a7ea6fde6a0d88ea4e7585f606eeb7f6c696ff2dcab138d26b276a47a7,3259 and tree 75111095142cc0f7bce196bc89c7494ba0576d0f63039a4953de6cda73ac38ba,4116
Expected: name == bloodhound, version == 0.17.0.0, tree == 75111095142cc0f7bce196bc89c7494ba0576d0f63039a4953de6cda73ac38ba,4116, cabal file == ab576560da2c4a87de631543f22e00cfec4ce545554220e4cf1033816b7695cd,3259

Expected behavior
Our stack should vendor the same version of hpack as upstream stack

Other solutions
Stack authors seem to agree that relying on hpack is bad for reproducibility precisely becuase it's unstable across releases https://tech.fpcomplete.com/blog/storing-generated-cabal-files

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[cdepillabout]

Hmm, this seems unfortunate.

I'm somewhat surprised you were able to figure this out. I don't know what I would have done had I just been presented with the hash for the cabal files differing. (Although having the problem explained to me, it does make sense what is happening.)

If you (or anyone else) wants to send a PR overriding stack's hpack dependency, I'll merge it in. I guess we should also make sure that stack in 19.09 and 20.03 are using the correct version of hpack.

[arianvp]

i'd love to but I have no idea how.
hackage-packages.nix states /* hackage-packages.nix is an auto-generated file -- DO NOT EDIT! */ but doesn't explain how i should generate it and from what, and how I then pin specific versions with it

Edit: found it https://github.com/NixOS/cabal2nix/tree/master/hackage2nix

[cdepillabout]

@arianvp

i'd love to but I have no idea how.

In general, you'd be editing the following file:

https://github.com/NixOS/nixpkgs/blob/29016450abebf5a1b7c423d977cbdf0b26d1597e/pkgs/development/haskell-modules/configuration-common.nix

Basically, you'd want to override the hpack dependency of stack so that it uses a different version of hpack.

Here's the existing modifications to the stack derivation:

nixpkgs/pkgs/development/haskell-modules/configuration-common.nix

Lines 1071 to 1098 in 2901645

	stack =
	let
	stackWithOverrides =
	super.stack.override {
	# stack-2.1.3.1 requires pantry-0.2.0.0.
	pantry = self.pantry_0_2_0_0;
	};
	in
	generateOptparseApplicativeCompletion
	"stack"
	(appendPatches stackWithOverrides [
	# This PR fixes stack up to be able to build with Cabal-3. This patch
	# can probably be dropped when the next stack release is made after
	# 2.1.3.1.
	(pkgs.fetchpatch {
	url = "https://github.com/commercialhaskell/stack/pull/5156.diff";
	sha256 = "0knk6f9fh1b4fxkhvx5gfrwclal4vi2va4zy34gpmwnjr7knf42y";
	excludes = [
	"snapshot-lts-12.yaml"
	"snapshot-nightly.yaml"
	"snapshot.yaml"
	];
	})
	# This patch fixes stack up to be able to build various GHC-8.8 changes.
	# This can hopefully be dropped when the next stack release is made
	# after 2.1.3.1 (assuming the next stack release uses GHC-8.8).
	./patches/stack-ghc882-support.patch
	]);

You can see that pantry is being overridden to a different version. You'd have to do something exactly like that, but for hpack.

You can find what versions of hpack are available with nix repl.

Checkout the nixpkgs repo and run nix repl:

$ cd /some/path/to/nixpkgs
$ nix repl ./.
nix-repl> haskellPackages.hpack<TAB>
haskellPackages.hpack          haskellPackages.hpack-convert  haskellPackages.hpack-dhall
nix-repl> haskellPackages.hpack.version
"0.33.0"

Looks like there isn't any version of hpack other than 0.33.0.

You'll have to add something to this list, so that a new hpack derivation is generated:

nixpkgs/pkgs/development/haskell-modules/configuration-hackage2nix.yaml

Line 2353 in 2901645

extra-packages:

See the following post for how this works:

https://discourse.nixos.org/t/haskellpackages-stm-containers-fails-to-build/5416/4

---

If you can't or don't want to figure out how to fix this, I recommend posting on the nixos discourse about this. There are lots of Haskell Nix users that like to help with stuff like this, but not everyone follows the issues here on GitHub.

edit: Oh, and there is the following post that talks about a couple different ways to figure out how Haskell stuff works here in nixpkgs:

https://discourse.nixos.org/t/call-to-action-for-updating-haskell-packages-after-bump-to-lts-15/6071

[simonmichael]

Already mentioned by @arianvp, but sol/hpack#381 would help this. hpack really is causing the trouble here.

[arianvp]

Stack 2.3.1 ships with a 'fix' that excludes packages that don't have cabal files from the lockfile commercialhaskell/stack#5210

[stale]

I marked this as stale due to inactivity. → More info

[cdepillabout]

We now ship the same version of hpack as upstream stack.

Feel free to open a new issue if you find that this is not the case.