Make Cabal v3 Nix-friendly

[michaelpj]

(I couldn't find an issue for this as a high-level thing, so I thought it was worth having one.)

There are a lot of awesome new features in Cabal v3, and I and many others are excited to use them.

However, I'm also a member of another group of users: those who make heavy use of Nix. For a number of reasons, Nix is very popular amongst the Haskell community, and nixpkgs builds a huge number of Haskell packages. And many of the industrial users are quite tightly wedded to it - for things like cross-compilation there are few compelling alternatives.

Now, most people using Nix to build Haskell packages do use Cabal. nixpkgs uses the lower-level Setup.hs machinery, but of course that still depends heavily on Cabal as a library.

So it would be great if Cabal v3 was awesome for Nix users too! I don't think this should be too hard - it mostly just means remembering it as a use case and a few key things like:

• Nix wants to be able to handle providing dependencies. Historically providing a populated package db has worked well for this.
• Nix builds happen in sandboxes that restrict things like network access, external folder access, etc.

Here are some existing issues that I think are relevant to Nix users:

• Add config flag to control generation of .ghc.environment files #4542 (fixed 🎉)
• Cabal new-build sometimes requires Internet access? #5783
• Cabal doesn't work when $HOME is not writable #4770 (maybe fixed?)
• document CABAL_CONFIG env variable. #5322 (fixed)

[phadej]

Hi Michael, thanks for you summary.

I'm however slightly confused. You say

nixpkgs uses the lower Setup.hs machinery

Yet all the issues is about cabal-install, the command line tool.

Let's not mix two things:

• nixpkgs dependencies derivations are built using Setup.hs, and this works well - there is no need to change to use cabal-install (old or new build) to build them.
• development workflow using new-build.
  • If you let cabal new-build to manage (Haskell) dependencies, everything works nicely.
  • But if you insist that (dependencies and) local packages are build using nix-derivations, there are no way to do it currently.

I assume you experience problems with the last point? Don't use generic-builder and cabal new-build together, they don't play well together.

Nix and cabal new-build clash in what they want to manage. The most niceties of cabal new-build is that cabal (the tool) manages the dependencies. For example if you add a dependency to your local package or try it with a different GHC version, the new dependencies will be calculated and installed etc. As far as I'm familiar with Nix, this changes would need you to amend your default.nix.

So there are two ways:

• Either simply get a nix-shell with cabal and ghc and let new-build do its stuff

• Or somehow convert cabal.project into Nix derivation(s) which:

  1. use cabal new-build --dry to generate install plan
  2. use something to generate expressions based on the install plan
  3. run some simple commands (i.e. not cabal new-build there anymore) in nix-shell.

  There are developments in that direction, for example @angerman's https://github.com/angerman/nix-tools
  But let Moritz expand about that

For comparison:

• https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-stack-builder.nix lets stack manage everything
• https://github.com/input-output-hk/stack2nix converts stack projects into nix expressions.

TL;DR we (well, nix users) need cabalProject2nix.

[angerman]

@phadej thanks for the ping. In fact we do use nix-tools in production. @michaelpj will soon as well (ha!). Not to derail this discussion much, but nix-tools does contain plan-to-nix, along side with stack-to-nix, as well as a completely rewritten builder that provides component level control.

[michaelpj]

@phadej thanks, this is very clarifying. Indeed, despite using Nix for building at scale we do also use cabal-install pretty heavily.

I want to highlight two cases: development workflow, and niceties of cabal

My typical development workflow when working on a package consists of:

• Enter a Nix shell to provide all the dependencies of the current package in a package db
• Use cabal to build/test the package that I'm working on

I've tried to replicate this across several packages using new-build, but as you say it wants to manage all the dependencies itself.

cabal is also just quite good at doing things that it would be nice to not have to replicate. For example, I have hopes that cabal new-haddock will evolve to produce a nice combined Haddock for the project (I just spend a few annoying days trying to create such a thing myself!). So it would be great to be able to use cabal new-haddock - and of course as a Nix user I'd like to put that in a derivation.

---

I think nix-tools might supersede the development flow (although I suspect Cabal will always be better at being incremental), but it would be a shame to have to invest time replicating things like the latter.

So here's a question: what would it take to persuade Cabal to accept some dependencies that I've already built? Can we transfer packages from a package db? Could we instead construct a store in the form that Cabal v3 expects and point it at it? (I believe that last point is already doable by setting $CABAL_DIR)

[hvr]

what would it take to persuade Cabal to accept some dependencies that I've already built?

It does already pick them up if they're registered in the global pkg-db; that's e.g. how cabal is able to use the pre-built packages provided by Linux distros (e.g. Debian has about 800 Haskell libraries from Hackage pre-built, and cabal tries to use them if possible!)

[michaelpj]

It does already pick them up if they're registered in the global pkg-db

Huh, great. I thought I had observed this not happening - let me see if I can reproduce that, and if so I'll report a bug.

[michaelpj]

Huh, great. I thought I had observed this not happening - let me see if I can reproduce that, and if so I'll report a bug.

Nope, can't reproduce, looks like I was doing something weird.

So it looks like things are in a better place than I thought they were, hooray! I think my only real complaint now is --offline not quite working.

[bgamari]

While perhaps not a cabal-install issue, the one aspect of the NixOS-cabal-install interaction that arguably could be improved here is the handling of native libraries. Currently I generally need to write a separate shell.nix for every project I work on to ensure that the needed native libraries can be found by new-build and then manually add include and library paths to cabal.project.local. I see two possible ways in which this could be improved:

• implement a nix expression which, given a Haskell package, would produce a shell environment with the native libraries depended upon by the package's transitive closure, or
• implement a pkg-config wrapper which would allow cabal-install to request the native libraries directly.

[bgamari]

For the record, quite a while ago I played around with the second option here but never was quite satisfied with the result.

[bgamari]

Prompted by this discussion and necessity, I sat down this afternoon to clean up the nix-pkgconfig hack mentioned earlier. Conceptually it's still rather hacky but in practice it is actually a fairly usable hack.

In short, it allows cabal-install to transparently resolve native package dependencies with nixpkgs derivations through use of a pkg-config wrapper and a pre-generated mapping from package names to nix attributes.

[hvr]

(I actually typed up this answer already 2 days ago but failed to send it)

@bgamari btw, your problem is yet another one which is not Nix specific at all, but occurs with other Linux distributions such as Debian-derived or Redhat-derived ones as well which provide packaged C libraries, and therefore this is a well known issue; and there's an old idea that never got implemented on how to address this which aims to extend the pkgconfig-depends capabilities by providing a mechanism for cabal to know not only which pkg-config entities are installed, but also about which pkg-config entities are procurable by the user even if they aren't installed just yet, as well as an operation to trigger materialising those pkg-config entities. Initially, the solver wouldn't even need to be made aware of the distinction between installed and available pkg-config items; you'd just have a flag to control whether the solver sees only the installed or also the virtual items. And then we'd only need to invoke the "procure pkgconfig-depends"-operation right before it's needed, i.e. shortly before configuring the first package that needs a not-yet installed pkgconfig-depends item.

At this point I'm starting to think that we don't need any Nix specific features at all in cabal; all problems I've heard so far about Nix-integration are merely specific instances of a more general problems that actually affect a larger target audience as well and therefore ought to be addressed by a slightly more general mechanism.

[bgamari]

Right.

Note that there is also another issue with the current incarnation of pkgconfig-depends: people tend to gate it on a flag for Windows compatibility. The two Haskell bindings which I most often run into (zlib and postgresql-libpq) both place the pkgconfig-depends dependencies behind a by-default-disable flag.

[hvr]

that's not a problem with the incarnation though but rather a problem with how people use it; ideally you'd rather have it behind an automatic flag where the branch using the pkgconfig-depend is the default one, so the solver can toggle it off -- as the other way round it'll never be toggled on;

However, there's something you wouldn't be able to express currently; and to address this case I wanted to see something like a "have(pkgconfig)" conditional:

-- strawman example
if have(pkgconfig)
    pkgconfig-depends: zlib >= 0.29.1
elif have(frameworks)
    frameworks: zlib
else
   extra-libraries: zlib

this would provide a way to be more explicit about which fallback strategy to use.

These have(<capability>) conditionals would also be useful in combination with #5328, e.g.

flag use_pkgconfig
   manual: True
   default: have(pkgconfig)

[rikvdkleij]

Does this issue also include that cabal new-haddock --enable-documentation does not work in the nix shell?

If so, it there a "workaround"?

Hmm, I just noticed that
cabal --enable-documentation --enable-nix new-haddock
works. So not running the command in the nix shell.

[schmittlauch]

#5783 should be kind of fixed once cabal-install-3.4 reaches nixpkgs: #4821 (comment)

But maybe we want to patch/ build the package in Nix with that option enabled by default?