Stack/Nix integration: does not include "git" in dependencies when going into nix-shell

[rrnewton]

To Reproduce:

git clone [email protected]:iu-parfunc/haskell-hpx.git
cd haskell-hpx/
git checkout 26fe7abfa61eca631fa2e278953c5767f9ce51be
stack build --verbose

Error output:

[debug] Run process: git clone https://github.com/RyanGlScott/bindings-dsl.git /home.local/rrnewton/working_copies/crest/haskell-hpx/.stack-work/downloaded/60aad9fdc2dc2a0455561cff67787ef91fbf8f7b54b9b2338446da12314249d5.tmp @(stack_GyRQQF9nCn6LxK1AXtTE8q:System.Process.Read src/System/Process/Read.hs:269:3)
2016-05-02 14:50:33.684448: [error] Executable named git not found on path:
["/nix/store/6l655f3376dv4wd2q94fffkjikyjajc4-hpx-2.2.0/bin","/nix/store/zzagkgsmb66gxng1819fyqrz7n08l7j4-ghc-7.10.3/bin","/nix/store/wz6508rdk04wn5nilzil1c7zbqg7lx4w-patchelf-0.9/bin","/nix/store/krwiax9xcn6m745k5ifwbcmavmhpi43k-paxctl-0.9/bin","/nix/store/jycz7ha7dm3ls1sjnznn3638193klbxv-gcc-wrapper-5.3.0/bin","/nix/store/xmrh23qgsbrk9gd0z72sj58kg1czmpcd-binutils-2.26/bin","/nix/store/d2ljpb414jpmzik3cvl0xlmr5m4aggp7-gcc-5.3.0/bin","/nix/store/bb32xf954imhdrzn7j8h82xs1bx7p3fr-glibc-2.23/bin","/nix/store/lphk3qrh53kv2j69b3sry5jmhfbrz1ns-coreutils-8.25/bin","/nix/store/lphk3qrh53kv2j69b3sry5jmhfbrz1ns-coreutils-8.25/bin","/nix/store/jshnvw5qxcbz8hyjx0k5g9f5ddlny7lb-findutils-4.4.2/bin","/nix/store/sfcxcsswbgbv2vfkf6ma420zikbz4k9n-diffutils-3.3/bin","/nix/store/c2rnzsrhby57x1cg9kab7zbcgcqfam1y-gnused-4.2.2/bin","/nix/store/rlw8bhavsn39fgvvxif89f4w476x6z7p-gnugrep-2.22/bin","/nix/store/wskqdq2cif8lnr59d1yy41j5k45n21k5-gawk-4.1.3/bin","/nix/store/h0cy1px0jdr0j331r79zr3i3bybwpxqw-gnutar-1.28/bin","/nix/store/4yl71n9qsjw5bkndzdg1aarzassy6hpy-gzip-1.6/bin","/nix/store/nlhixpi844nri62cdkcp8d4dkhxwvj2j-bzip2-1.0.6/bin","/nix/store/cqnrdrxkwbq835rkyxcscdw119yppjpl-gnumake-4.1/bin","/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin","/nix/store/n1ih0w9qcjswvprycxxgx82wg10h8dpj-patch-2.7.5/bin","/nix/store/nbi1p1l7f25nr8ji9vf6bfi4blw9dhwa-xz-5.2.2/bin"]

The problem seems to be with the nix-shell command that stack is formulating, which does not include git:

Run process: nix-shell --pure -I nixpkgs=https://github.com/iu-parfunc/nixpkgs/archive/hpx-master.tar.gz -E "wi th (import <nixpkgs> {}); runCommand \"myEnv\" { buildInputs=lib.optional stdenv.isLinux glibcLocales ++ [ hpx haskell.packages.lts-5_9.ghc ]; STAC K_PLATFORM_VARIANT=''nix''; STACK_IN_NIXSHELL=1; STACK_IN_NIX_EXTRA_ARGS='' --extra-lib-dirs=${hpx}/lib --extra-include-dirs=${hpx}/include --extr a-lib-dirs=${haskell.packages.lts-5_9.ghc}/lib --extra-include-dirs=${haskell.packages.lts-5_9.ghc}/include '' ; } \"\"" --command "'/nix/store/kf f433s61k9808vwb9m0yl8pmipirvxs-stack-1.0.4.3/bin/stack' $STACK_IN_NIX_EXTRA_ARGS '--internal-re-exec-version=1.0.4.3' 'build' '--verbose'" @(stack_ HKArHFFZq7p7BGX6AeJtxh:Stack.Exec src/Stack/Exec.hs:51:5)

So the solution would seem to be to either (1) always ask for git, or (2) ask for git only when the stack.yaml file includes stanzas like this:

- location:
    git: https://github.com/RyanGlScott/bindings-dsl.git
    commit: e918a9eea5b153c78f4a95c30262fdf917603d2c

CC @RyanGlScott

[rrnewton]

If I manually add git into the nix packages in that stack.yaml file, the next error is this:

    Configuring bindings-libffi-0.3...
    setup-Simple-Cabal-1.22.5.0-ghc-7.10.3: The program 'pkg-config' version
    >=0.9.0 is required but it could not be found.

If Cabal needs pkg-config, shouldn't that one also be stacks responsibility to include in the dependencies?

[YPares]

@rrnewton Actually yes for now it's something you should add manually to your nix packages when you need it.
Maybe we should add something to the nix doc if it trips people.

[mboes]

Surely the docs can be improved yes, since it tripped up @rrnewton. The question is - is there a bug here, no matter how easy the workaround?

There is a tension between automagic and least surprise. If Stack silently adds packages inside a sandbox, then the content of the sandbox is ill-defined, and not easy to reproduce when e.g. the user switches to using a shell-file line rather than a packages line. I think that if Stack needs git in the sandbox, the user should say that Git is needed in the sandbox. If pkg-config is needed for the compile, the user should say so too. If the user wants some custom git with a different package name, the user should be able to specify that too, without conflicts arising from silently adding things into the sandbox. So IMHO there's a docfix here, and also a helpful error message to add to suggest a good fix to the user, but I'm skeptical there's an implementation change worth doing.

[rrnewton]

@mboes : I agree that there's a tension here. IMHO there is necessarily a degree of auto-magic in this feature. Stack is launching itself inside Nix! Am I supposed to know that as a user? I.e. that stack calls itself, via e.g. /nix/store/kf f433s61k9808vwb9m0yl8pmipirvxs-stack-1.0.4.3/bin/stack? Or is that an implementation detail that is below the level of abstraction I need to think about?

If stack is automagically including itself in the nix derivation (not to mention haskell.packages.lts-5_9.ghc, which the user didn't explicitly include either), shouldn't it include any other dependencies it needs for internal stack business? That is, for subprocesses that stack will launch. How can the user know whether stack is implemented so as to shell out to git? What about wget or curl? What else might stack shell out to?

Btw, in the above example I think it's Cabal that brings in the pkg-config dependency.

Regarding the possibility of "conflicts arising": it's really too bad that the current way of calling into nix-shell doesn't provide a means for stack to have a precise dependency on git which is not visible to the user. Do we simply need to add git as a runtime dependency in the derivation that builds haskellPackages.stack? I'm not a Nix expert, so I don't really know how this should be formulated.

[YPares]

Stack is not adding itself in the nix derivation, it relaunches the same
executable inside the nix-shell.
Only GHC is from the nix store.

2016-05-03 17:12 GMT+02:00 Ryan Newton [email protected]:

@mboes https://github.com/mboes : I agree that there's a tension here.
IMHO there is necessarily a degree of auto-magic in this feature. Stack is
launching itself inside Nix! Am I supposed to know that as a user? I.e.
that stack calls itself, via e.g. /nix/store/kf
f433s61k9808vwb9m0yl8pmipirvxs-stack-1.0.4.3/bin/stack? Or is that an
implementation detail that is below the level of abstraction I need to
think about?

If stack is automagically including itself in the nix derivation (not
to mention haskell.packages.lts-5_9.ghc, which the user didn't explicitly
include either), shouldn't it include any other dependencies it needs for
internal stack business? That is, for subprocesses that stack will launch.
How can the user know whether stack is implemented so as to shell out to
git? What about wget or curl? What else might stack shell out to?

Btw, in the above example I think it's Cabal that brings in the pkg-config
dependency.

Regarding the possibility of "conflicts arising": it's really too bad that
the current way of calling into nix-shell doesn't provide a means for
stack to have a precise dependency on git which is not visible to the
user. Do we simply need to add git as a runtime dependency in the
derivation that builds haskellPackages.stack?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#2089 (comment)

[mgsloan]

@rnewton makes a good point about

What else might stack shell out to?

If we want full reproducibility for container builds, we can't have stack depend on more external programs in the future. Seems like there are a few decent ways of handling this:

• Simply avoid adding such deps. This is mostly what's happened, I don't think we've added nearly any mandatory external deps.
• Automagic discussed here. Seems reasonable if put under a flag, but is it worth the implementation effort / maintenance?
• Lazy Automagic - Wait until we need an additional external deps and solve the problem then. Need to remember to consider this, though. Could be easy to forget, particularly if it's a usage of a program that's very commonly installed.

[rrnewton]

@YPares
Oh, I see. It must be that I just happened to use a stack from the nix store in this case. Hmm, reexecuting the same stack binary requires a bit of impurity then? If it actually executed under a chroot environment, then the original stack executable wouldn't be able to find its dynamically-linked libs, right?

I admit to being a bit confused by nix-shell -- it seems impure compared to the real nix-build environments. It includes junk like your .bashrc, and I was having trouble with PKG_CONFIG_PATH coming through and breaking the build.

[rrnewton]

Ok, it sounds like folks in this thread would perhaps prefer a patch that implements something like this:

• detect binary not found for binary \in { git, pkg-config } (possibly more later)
• issue warning to user, e.g.: "stack needs access to git in PATH, please add git to your list of packages or shell.nix environment."

The implication is that the ability to shell out to git/pkg-config becomes part of the "public interface" of stack/Cabal, not an internal implementation detail hidden by the abstraction. As people here have suggested, as long as that set of things does not grow, that's probably not too bad for users to learn.

[rrnewton]

I want to understand this previous question though, before letting it drop:

@YPares : regarding stack not adding itself to the derivation -- why not? Then wouldn't it be able to list things like git and pkg-config as runtime dependencies in a precise way?

To possibly answer my own question, something seems to be wrong with haskellPackages.stack. If I do a "copy-closure" operation, its huge, >1GB. It seems like it is incorrectly roping in compile dependencies (to build stack) as runtime dependencies as well.

[YPares]

Because I see no point in making the user re-dowload a software they
already have for no added benefit, and also more importantly you're not
guaranteed that the nix packages contain stack at the exact same version
that the system-installed stack.

2016-05-04 14:49 GMT+02:00 Ryan Newton [email protected]:

I want to understand this previous question though, before letting it drop:

@YPares https://github.com/YPares : regarding stack not adding itself
to the derivation -- why not? Then wouldn't it be able to list things like
git and pkg-config as runtime dependencies in a precise way?

To possibly answer my own question, something seems to be wrong with
haskellPackages.stack. If I do a "copy-closure" operation, its huge,

1GB. It seems like it is incorrectly roping in compile dependencies (to
build stack) as runtime dependencies as well.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2089 (comment)

[rrnewton]

Yes, to get a matched version nixpkgs would need to contain at least all the publicly released versions of stack. That way stack could ask for the same version as its currently running.

When using either Nix or Docker, it seems like massive downloads and disk usage are in the cards ;-). I don't think the extra 10MB for another stack binary would be a big problem.

@mboes: I read the original blog post. Is there an explanation somewhere of why the design has stack shelling into nix-shell rather than generating a nix expression and using nix-build?

[YPares]

stack remains the build tool.
nix-build is not. It doesn't cache build artifacts and puts everything in
the nix store everytime a build succeeds. It's not made for iterative
development.

2016-05-04 15:33 GMT+02:00 Ryan Newton [email protected]:

Yes, to get a matched version nixpkgs would need to contain at least all
the publicly released versions of stack. That way stack could ask for the
same version as its currently running.

When using either Nix or Docker, it seems like massive downloads and disk
usage are in the cards ;-). I don't think the extra 10MB for another stack
binary would be a big problem.

@mboes https://github.com/mboes: I read the original blog post. Is
there an explanation somewhere of why the design has stack shelling into
nix-shell rather than generating a nix expression and using nix-build?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2089 (comment)

[rrnewton]

To be honest, the current integration is fragile in ways I don't yet understand.

For instance, if you download that repo mentioned at the top of this issue, it builds with make build-manual but fails with make build-nix. The former calls stack build explicitly within nix-shell, the latter calls stack build outside the shell, and lets stack formulate the call to nix-shell. The output on one of our systems is shown here in this gist.

As far as I can tell, they really should do the same thing. The hwloc dependency it's choking on is included explicitly in the buildInputs in the manual case, but it's successfully stuck into buildInputs of the nix expression generated by stack and passed to nix-shell -E as well...

[rrnewton]

@YPares: Thanks, I can see that if nix-build must always create an output to /nix/store, then it is not suitable. I was just a little surprised that nix-shell is so impure compared to nix-build (even with --pure!). Sourcing the .bashrc undermines reproducible builds, and it doesn't chroot.

I suspect this means that stack/Docker will provide stronger isolation and better reproducibility than stack/nix for now.

[mboes]

@mboes: I read the original blog post. Is there an explanation somewhere of why the design has stack shelling into nix-shell rather than generating a nix expression and using nix-build?

@rrnewton I'm not sure I have a good answer to that question. nix-shell seems to be what the Nix folks typically use to create a local environment with local dependencies, so that's what we went with. It's not meant to be properly isolated: it's just a local env providing tools locally that you can use to do anything you like on any file you like when shelling in. I agree that if you prefer proper isolation then Stack's Docker support will give you much stronger guarantees. It just requires you jumping through slightly more hoops when you do want to poke holes through the isolation layer, e.g. launching a local server you want to make accessible to the host etc.

[mboes]

Ok, it sounds like folks in this thread would perhaps prefer a patch that implements something like this:

• detect binary not found for binary \in { git, pkg-config } (possibly more later)
• issue warning to user, e.g.: "stack needs access to git in PATH, please add git to your list of packages or shell.nix environment."

@rrnewton sounds eminently reasonable to me. Let's open a new ticket regarding that.