Haskell mkDerivation env attribute sucks in the source

[twhitehead]

Issue description

I'm using a setup based on the nixpkgs manual haskell guide. Since switching to the 18.09 package set, nix-shell now dumps my development directory to the store every time it is run.

This adds significant shell start up time and consumes a lot of unnecessary disk space by constant snapshotting my work.

I believe the issue is that the env attribute of the haskell mkDerivation function now sucks in the source specified inmkDerivation, which is set to ./. by cabal2nix . when following the haskell development instructions in the nixpkgs manual.

Steps to reproduce

Here is a small example of following the directions (where we are using lub to simulate our own package that we are developing)

wget -d http://hackage.haskell.org/package/lub-0.1.7/lub-0.1.7.tar.gz
tar xzf lub-0.1.7.tar.gz
cd lub-0.1.7
cabal2nix . > lub.nix
cat > default.nix <<"EOF"
nixpkgs ? import <nixpkgs> {}, compiler ? "ghc843" }:
nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./lub.nix { }
EOF
cat > shell.nix <<"EOF"
nixpkgs ? import <nixpkgs> {}, compiler ? "ghc843" }:
(import ./default.nix { inherit nixpkgs compiler; }).env
EOF
dd if=/dev/zero of=big bs=$((1024*1024)) count=500
nix-shell -I nixpkgs=/home/tyson/.nix-defexpr/channels/nixpkgs

warning: dumping very large path (> 256 MiB); this may run out of memory

Technical details

Here is the requested info. Note that the machine I'm currently on is 18.03, but I'm running against the 18.09 channel for the development that I'm currently doing.

• system: "x86_64-linux"
• host os: Linux 4.16.17, NixOS, 18.03.132768.94d80eb7247 (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.0.4
• channels(tyson): "nixpkgs-18.09.1206.6d6e4e6b558"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[twhitehead]

@infinisil @ElvishJerricco @peti ccing you guys as you all show up in the log around the new haskellPackages.shellFor code.

[peti]

The env attribute most definitely won't refer to ./.. That attribute has no references outside of nixpkgs. Now, if you instantiate env for a package that you've defined locally, then the definition is yours; you are free to add all kinds of filters to ignore non-relevant files from ./., i.e. using stdenv.lib.cleanSource or stdenv.lib.cleanSourceWith. However, that's your responsibility; Nixpkgs can't do that for you.

[twhitehead]

Let me restate the issue. Should env reference the src attribute? It seems in 18.03 it didn't and now in 18.09 it does. I would think the old behaviour is correct and the new one wrong as env is about an environment with the depended upon packages and not the source.

[peti]

env does not reference src. Here is proof:

/tmp $ cabal get lub
Downloading  lub-0.1.7
Downloaded   lub-0.1.7
Unpacking to lub-0.1.7/

/tmp $ cd lub-0.1.7

/tmp/lub-0.1.7 $ cabal2nix --shell cabal://lub >shell.nix

/tmp/lub-0.1.7 $ dd if=/dev/zero of=big bs=$((1024*1024)) count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.119223 s, 4.4 GB/s

/tmp/lub-0.1.7 $ nix-shell --run zsh
these derivations will be built:
  /nix/store/r5flaqch6g99ms9cn5lmcy6qp2iganla-ghc-8.4.4-with-packages.drv
these paths will be fetched (1.59 MiB download, 8.90 MiB unpacked):
  /nix/store/60dp36l89ga5wrdfp1pp8srcsgdnjn4v-bash-interactive-4.4-p23-man
  /nix/store/c3rmm8haigs4j9yv1kmxkwys9yylmnmq-unamb-0.2.7
  /nix/store/czdrkg5m2avxjclyrfmyj7jdg04jqmgn-unamb-0.2.7-doc
  /nix/store/gahjzf8ii69s6fnyp1pjbxsdsd8zcqpj-bash-interactive-4.4-p23-dev
  /nix/store/rbw2vhaqs79a3f3vvmky7cvyjdhaa8mv-bash-interactive-4.4-p23-info
  /nix/store/rr09yd980jjbx9q4cfnwwzhax3q8za8s-bash-interactive-4.4-p23-doc
  /nix/store/sik5ydsar56zw8mm4ygdclpcz9hq16ia-readline-7.0p5
  /nix/store/sncyhsg7zc06hc31lncj4hsy3dh5wmi1-bash-interactive-4.4-p23
copying path '/nix/store/rbw2vhaqs79a3f3vvmky7cvyjdhaa8mv-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/rr09yd980jjbx9q4cfnwwzhax3q8za8s-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/60dp36l89ga5wrdfp1pp8srcsgdnjn4v-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/sik5ydsar56zw8mm4ygdclpcz9hq16ia-readline-7.0p5' from 'https://cache.nixos.org'...
copying path '/nix/store/czdrkg5m2avxjclyrfmyj7jdg04jqmgn-unamb-0.2.7-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/sncyhsg7zc06hc31lncj4hsy3dh5wmi1-bash-interactive-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/gahjzf8ii69s6fnyp1pjbxsdsd8zcqpj-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/c3rmm8haigs4j9yv1kmxkwys9yylmnmq-unamb-0.2.7' from 'https://cache.nixos.org'...
building '/nix/store/r5flaqch6g99ms9cn5lmcy6qp2iganla-ghc-8.4.4-with-packages.drv'...

/tmp/lub-0.1.7 $

big isn't copied.

[twhitehead]

While I work to follow what your response above, I offer you this counter challenge

cat example.nix

with import <nixpkgs> { };

let
  example = { mkDerivation, stdenv, base, linear }: mkDerivation {
    pname = "example";
    version = "0.0.1";
    src = throw "what does the current source have to do with env?";
    libraryHaskellDepends = [ base linear ];
    license = stdenv.lib.licenses.bsd3;
  };

in
  (haskellPackages.callPackage example { }).env

On my 18.03 based desktop

[tyson@tux:~]$ nix-shell example.nix 
[nix-shell:~]$ 

on my 18.09 laptop

[tyson@manny:~]$ nix-shell example.nix 
error: what does the current source have to do with env?
(use '--show-trace' to show detailed location information)

[tyson@manny:~]$ nix-shell --show-trace example.nix 
error: while evaluating the attribute 'NIX_GHC' of the derivation 'ghc-shell-for-example-0.0.1' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating 'ghcWithPackages' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:219:23, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:275:18:
while evaluating 'withPackages' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:116:18, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:219:35:
while evaluating 'callPackageWith' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/customisation.nix:108:35, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:116:28:
while evaluating 'makeOverridable' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/customisation.nix:67:24, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/customisation.nix:112:8:
while evaluating anonymous function at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/with-packages-wrapper.nix:1:1, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/customisation.nix:69:12:
while evaluating 'closePropagation' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:228:22, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/with-packages-wrapper.nix:45:57:
while evaluating 'uniqList' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:159:14, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:228:29:
while evaluating 'go' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:160:18, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:166:8:
while evaluating 'innerClosePropagation' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:211:32, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/deprecated.nix:228:52:
while evaluating anonymous function at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:275:32, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:219:49:
while evaluating anonymous function at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:270:12, called from undefined position:
while evaluating anonymous function at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:270:33, called from /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/development/haskell-modules/make-package-set.nix:270:19:
while evaluating the attribute 'outPath' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/lib/customisation.nix:150:7:
while evaluating the attribute 'src' of the derivation 'example-0.0.1' at /nix/store/l9hcs7gapmg0d0hizi146h9mn03774ga-nixos-18.09/nixos/pkgs/stdenv/generic/make-derivation.nix:177:11:
what does the current source have to do with env?

[twhitehead]

Okay. I followed your example through. The shell.nix file generated as you did above doesn't reference ./. and building it doesn't suck any of it in as you would expect. I understand your point to be that the current directory was only sucked in because it was referenced in my .nix file.

I don't disagree with this. What I am saying is when you used to use the .env attribute, it seemed to totally ignore the src variable. Now it seems to suck it in despite (at least to my understanding) not needing to. You can see this above in my example where I set src to throw an exception. Under 18.03 no exceptions are raised. Under 18.09 it explodes in your face.

Strangely, in my testing, whether it explodes or not seems to depend on exactly what libraries you include. If you include just base it is okay. If you include bytestring it is okay. If you include unamb or linear it explodes in your face. Seems a bit strange.

Thanks! -Tyson

[infinisil]

After bisecting it was clear that my #46453 was the cause of this, now trying to figure out why..

[infinisil]

Aha! I found the culprit. It's these lines originally introduced by @ElvishJerricco in #36393 (comment):

nixpkgs/pkgs/development/haskell-modules/make-package-set.nix

Lines 272 to 277 in 483914b

	# If `packages = [ a b ]` and `a` depends on `b`, don't build `b`,
	# because cabal will end up ignoring that built version, assuming
	# new-style commands.
	haskellInputs = pkgs.lib.filter
	(input: pkgs.lib.all (p: input.outPath != p.outPath) selected)
	(pkgs.lib.concatMap (p: p.haskellBuildInputs) packageInputs);

It's to filter out dependencies that are in the selected package list, because those usually want to get build by cabal directly (for development), not through nix.

The input.outPath != p.outPath comparison requires both input and p to be evaluated as derivations, which means strictly. And because p comes from selected which comes from packages which comes from the package you use the .env of, the source of it gets evaluated.

Now how could we solve this:

• Do a smarter filter that doesn't need to compare derivations a with a's direct dependencies (because those will never be equal)
• Just change the filter to a name comparison (input.name != p.name)

[infinisil]

This also explains why it didn't blow up with bytestring and base: These libraries are packaged with ghc, so they're really null (check with nix eval nixpkgs.haskellPackages.base), which means that the helper functions won't be able to determine any dependencies for these, so haskellBuildInputs of those will be [], meaning concatMap (p: p.haskellBuildInputs) packageInputs = [], so the filter doesn't even call the function doing the strict derivation thing because there are no elements to filter.

[ElvishJerricco]

@infinisil I think filtering on the name field is a fine alternative. A little disappointing that we don't get the nice exactness, but it'll be fine. Alternatively, we could do the outPath comparison after using overrideCabal to set src to null on both packages.

[peti]

@twhitehead, you were right! Nice catch. Thank you for following through with the bug report.

[twhitehead]

@peti no problem. Reading back through my original error report, I can see it wasn't the greatest for really bringing out the issue. I really should have led with the second throw example. Thanks for the original followup that got me to clarify things.

[ElvishJerricco]

So I've tested this both ways: comparing pname, and comparing overrideCabal x (_: { src = null; }), and both work fine. @peti do you have an opinion on which way we should go? The former is less precise, but the latter may suck in extraneous sources if anything is manually set as environment variables or something.

[infinisil]

@ElvishJerricco I think the src = null method is better. It will cover 99.9% of usecases, and doesn't have any significant downside. The pname way could lead to unexpected results.

[twhitehead]

Thanks! 👍

[grahamc]

Reopening because the related commit was reverted in 18.09 in a1156b9, because the tarball job was failing.

[grahamc]

Funny that revert commits also close issues :)

[peti]

I am unsure about the state of this issue. Is this still relevant?

[ElvishJerricco]

@peti Not totally sure, but it seems fixed:

(with import (fetchTarball channel:nixos-19.09) {}; haskellPackages.mkDerivation {
  pname = "foo";
  version = "1";
  license = null;
  src = ./foobarbazbar; 
}).env

$ mkdir ./foobarbazbar
$ nix-shell foo.nix --run :
$ find /nix/store -maxdepth 1 -name "*-foobarbazbar"

The find command generates no output, implying no directory named foobarbazbar was sucked in. Not sure if this is comprehensive or not.

[infinisil]

@ElvishJerricco See #51079 (comment), I can still reproduce it with that example there

[twhitehead]

I haven't done any further looking into this since the reversion, but, at least in the original issue, you need a dependency outside of base to trigger it, such as linear in the above example. The reason for this at that time was explained in this comment, and the one above it, by @infinisil.

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[twhitehead]

Just finished checking my earlier test case (#51079 (comment)) and pleased to report there is no exception thrown on 20.03. I'll close this issue. Thanks everyone for the work! 😄

[expipiplus1]

This has reoccurred, in fact I opened a duplicate issue I think some months ago: #86775

I also have a minimal reproducer https://github.com/expipiplus1/nix-repro-86775