Improve stack workflow/UX with haskell.nix

[codygman]

• update the docs with a minimal copy-pastable example for stack (moving extra stuff to "appendix" or something)
• figure out if we want to work with stack to disable their nix support when in nix-shell (NIX_SHELL=impure)

This is a WIP proposal of sorts, but is trying to record learnings from hacking around with haskell.nix's stack support during ZuriHac 2020. I think the ideal UX for a stack user starting to use haskell.nix would be.

Create a new stack project:

stack new myproj

Add a default.nix:

let 
  haskellNix = import (builtins.fetchTarball https://github.com/input-output-hk/haskell.nix/archive/master.tar.gz) {};
  nixpkgsSrc = haskellNix.sources.nixpkgs-2003;
  nixpkgsArgs = haskellNix.nixpkgsArgs;
in
{ pkgs ? import nixpkgsSrc nixpkgsArgs
, haskellCompiler ? "ghc883"
}:
pkgs.haskell-nix.stackProject {
  src = pkgs.haskell-nix.haskellLib.cleanGit { name = "myproj"; src = ./.; };
  compiler-nix-name = haskellCompiler;
}

Build it with haskell.nix:

nix build -f . myproj.components.library

That takes care of building your binaries, but then what about general development? Ideally I think you would do:

nix-shell -A shellFor
stack ghci

In practice though, this doesn't work out. I'm working on creating a repo with a readme that logs what happens when you try this.

[codygman]

The problem is that if you're already inside of a nix-shell which provides all the libraries stack needs you need to disable stack's own nix integration like:

[nix-shell:~/code/tmp/haskell-nix-stack-workflow]$ stack --stack-yaml stack-nix.yaml --no-nix ghci

[codygman]

So things work very well and seem to be seamless as of this working tree.

[nh2]

I've also updated now my comment on commercialhaskell/stack#3617 (comment) to include the --no-nix so that it works on NixOS, and explained why it's needed.

[tscholak]

maybe relevant: @cdepillabout devised a stack+nix workflow for hasktorch, see this PR: hasktorch/hasktorch#382

[codygman]

@tscholak Thanks! From a super quick glance, I think that uses stack's nix integration whereas this method circumvents and turns off stack's nix integration. Perhaps when @cdepillabout get's time they will have useful comments on their approach vs ours 🙂 .

[cdepillabout]

@tscholak Maybe I didn't quite understand the original message, but why would you turn off stack's nix integration?

In my experience it works pretty well, and it doesn't require you to jump into a nix-shell before using it.

The solution in hasktorch/hasktorch#382 is the same approach we've been using at work as well for a while now, and it seems to work alright (aside from the various problems of using stack with nix).

[codygman]

Maybe I didn't quite understand the original message, but why would you turn off stack's nix integration?

Basically to use it as a "dumb" interface to the packages nix has already put in place.

In my experience it works pretty well, and it doesn't require you to jump into a nix-shell before using it.

I recall having at least one issue with the stack nix integration (will have to search my notes) and it seemed better to lean more on nix than stack's nix integration. I think it's also closer to how the cabal integration works, but that's only a feeling.

In my experience it works pretty well, and it doesn't require you to jump into a nix-shell before using it.

Ah, that's what I've been using direnv for and my end goal is "cd into directory and ghcide, stack ghci, stack build work transparently" and avoid "create a stack wrapper script for ghcide" or "customize emacs to call stack first". I think your solution using the nix integration might also accomplish all if not most of these though.

[cdepillabout]

@codygman To be honest, I'm not sure if there is a strong reason to prefer one way over the other.

As you say, using stack --nix requires one less call (you don't have to worry about a nix shell environment since stack handles it for you), but that doesn't sound like it matters to you if you use direnv or something like that.

Also, the upstream stack documentation recommends stack --nix, and doesn't talk about sidestepping the nix integration. Personally, I'd prefer to follow the recommendations by the stack developers, just so it makes it easier to get help and report bugs when things go wrong.

I haven't used ghcide, but I know getting tooling working is sometimes tough. Without having used ghcide, I'm not sure what the best approach is here.

[domenkozar]

I've addressed the first bullet point from this issue in #702

There's a TODO in Getting Started tutorial to specify how to use stack repl with haskell.nix provided packages.

[codygman]

@cdepillabout Does your stack --nix integration in hasktorch/hasktorch#382 use haskell.nix? It looks like so, but I saw some note in the readme that suggests stack is used directly for Haskell dependencies.

I was hoping to use stack as a sort of light ui, use haskell.nix to translate the package.yaml to nix expressions, then when stack starts up it simply does nothing but start up a stack build using the packages nix already got for it.

That way we benefit from cachix storing the system and haskell dependencies.

Edit: guessing no unless stack output for nix integration is the exact same:

[cody@nixos:~/code/hasktorch]$ stack --nix build
libtorch-ffi    > configure (lib)
libtorch-ffi    > Configuring libtorch-ffi-1.5.0.0...
JuicyPixels     > configure
...

[codygman]

And since it just happened... to avoid stack errors building the haskell files like:

... snip ...
aeson           > /run/user/1000/stack-d05642f5884b8e7a/aeson-1.4.7.0/Data/Aeson/Text.hs:31:1: error:
aeson           >     Bad interface file: /home/cody/.stack/snapshots/x86_64-linux-nix/e5d9b996e75045474bbff7b6146ec243542c1d8876170c4eccdcb1902937f59a/8.8.3/lib/x86_64-linux-ghc-8.8.3/scientific-0.3.6.2-88AAZ2094ShCvz9CIiUOKQ/Data/Text/Lazy/Builder/Scientific.hi
aeson           >         /home/cody/.stack/snapshots/x86_64-linux-nix/e5d9b996e75045474bbff7b6146ec243542c1d8876170c4eccdcb1902937f59a/8.8.3/lib/x86_64-linux-ghc-8.8.3/scientific-0.3.6.2-88AAZ2094ShCvz9CIiUOKQ/Data/Text/Lazy/Builder/Scientific.hi: openBinaryFile: does not exist (No such file or directory)
aeson           >    |                                     
aeson           > 31 | import Data.Text.Lazy.Builder.Scientific (formatScientificBuilder)
aeson           >    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
....

Then again I could be responsible for that and maybe it's not stack related. So this is a possible data point pointing towards relying on stack doing haskell installs not being as ideal depending on whether I caused that error or not.

[codygman]

However, a downside is stack doesn't expect to be used this way. For instance I'd prefer if there was a flag like stack --never-install-packages because I'm currently facing a weird issue where something my haskell.nix -> nix-shell -> stack ghci pipeline isn't installing Esqueleto and stack decides it should download it.

That leaves me in a state where most things are installed with nix, some installed with stack. While it's not the worst thing in the world, it's not ideal, and puts a big damper on the entire point imo.

[codygman]

I'm running into a very interesting issue where it seems that any Haskell dependency with a native dependency like postgresql-libpq isn't actually present in the nix-shell according to ghc-pkg list.

I'll work on making a reproduction in my linked repo and submit another issue if that's the case. Hopefully if this bug actually is present, it's not just failing silently somehow.

[cdepillabout]

@codygman

Does your stack --nix integration in hasktorch/hasktorch#382 use haskell.nix?

Not really.

When you use Stack's Nix integration, there is no (easy) way to get Haskell packages compiled by Nix. Stack really wants to compile all the Haskell packages from scratch.

If you want to get all the Haskell dependencies compiled by Nix, I'd recommend just using a cabal-based workflow, since it is able to use a package database built by Nix.

[codygman]

In regards to my issue above, I couldn't reproduce with the simple example sadly 😞

After this change:

diff --git a/package.yaml b/package.yaml
index 9d5ca87..4550416 100644
--- a/package.yaml
+++ b/package.yaml
@@ -22,6 +22,7 @@ description:         Please see the README on GitHub at <https://github.com/gith
 dependencies:
 - base >= 4.7 && < 5
 - postgresql-simple
+- persistent-template

This test passed:

nix-shell --pure --run 'echo -e "\nis persistent in package.yaml?"; grep persistent package.yaml; echo -e "\nIs persistent now in ghc package db?"; ghc-pkg list | grep persistent'

[codygman]

@cdepillabout

Does your stack --nix integration in hasktorch/hasktorch#382 use haskell.nix?

Not really.

When you use Stack's Nix integration, there is no (easy) way to get Haskell packages compiled by Nix. Stack really wants to compile all the Haskell packages from scratch.

This repo uses stack after downloading Haskell packages from nix and things appear to work flawlessly (in the very simple case at least).

I tried migrating this approach to a larger repo for work though and there are some harder to debug issues I have to isolate, though it still works nearly all the time on NixOS, almost as often on Ubuntu+nixpkgs, and likely similar on MacOS+nixpkgs but target/lorri#362 is blocking testing there for me.

It seems to be with packages that have native dependencies or silent errors about packages not in the dependency set. Not sure.

For this approach to be truly robust though, there'd need to be a stack --never-install flag, then if nix didn't provide the full package set needed stack wouldn't go ahead and try to set them up anyway.

If you want to get all the Haskell dependencies compiled by Nix, I'd recommend just using a cabal-based workflow, since it is able to use a package database built by Nix.

I tried that, but my team wants to stay with package.yaml and that doesn't integrate so nicely with lorri. So... this seems the path of least resistance with the added bonus that my team could keep using stack transparently with nix under the hood with all it's benefits.

That's what I'm working towards here anyway 😄

[cdepillabout]

This repo uses stack after downloading Haskell packages from nix and things appear to work flawlessly (in the very simple case at least).

I tried migrating this approach to a larger repo for work though and there are some harder to debug issues I have to isolate

This is why I added the "(easy)" qualifier.

While it is possible in some ways, it is not the easy path when using stack+nix.

If you're going to venture off the easy path, you can expect to run into some problems.

my team wants to stay with package.yaml

I may have linked it to you recently, but Snoyman has been recommending people not go solely with package.yaml anymore: https://www.fpcomplete.com/blog/storing-generated-cabal-files/

[codygman]

Stack apparently has an --only-locals flag which will fail if the package isn't available locally because nix failed for some reason. I wonder how the error message will be. Related stack discussion at commercialhaskell/stack#5272

Edit: opened an issue in stack at commercialhaskell/stack#5319 since this only applies to the stack build command and we'd like it to apply to all stack commands.

[codygman]

@cdepillabout I just encountered one _potential_answer to "why disable stack's nix integration".

$ [cody@nixos:~/code/ghcide]$ stack build --fast --nix  && stack exec --nix -- ghcide /tmp/blah/Bug.hs 
...
  Failed to parse result of calling stack
  UnliftIO.Exception.throwString called with:
  
  In Nix shell but reExecL is False

That will lead you to commercialhaskell/stack#5000 where @snoyberg says:

Both the Docker and Nix codepaths (which both leverage the same reExecL feature) are notoriously susceptible to breaking in weird ways when their environment is modified in unexpected ways. We've tried to lock things down to ensure that they are run in normal environments. The exact thing you're trying to accomplish may run just fine, but it's untested, and I'd rather encourage people towards using tested codepaths than try to figure out exceptions we can start adding.

I've not been able to figure out how to fix this issue and I suspect it's something that pure nix/dumb stack solution would fix whereas the default smart stack/dumb nix solution falls down.

[cdepillabout]

@codygman Ah, that's an interesting catch. I've also run into this reExecL thing before as well, and it is quite annoying.

However, I don't quite understand why you're running into it here. It doesn't look to me like stack is being re-exec'd anywhere between those two stack build and stack exec commands. Do you know if ghcide internally calls stack?

[codygman]

@codygman Ah, that's an interesting catch. I've also run into this reExecL thing before as well, and it is quite annoying.

However, I don't quite understand why you're running into it here. It doesn't look to me like stack is being re-exec'd anywhere between those two stack build and stack exec commands. Do you know if ghcide internally calls stack?

Yep, that's it. It calls stack or cabal internally.

[cdepillabout]

@codygman Hmm, if ghcide knows how to run stack, would you be able to do something like stack build --fast --nix && ghcide /tmp/blah/Bug.hs.

[domenkozar]

Note that commercialhaskell/stack#5000 was fixed in stack master.