The future of all-hies

[infinisil]

Here's a project update, what's been happening and what the plan is for the future. This addresses #60.

What's happened?

Disregarding the issues with Cabal and glibc, everything worked decently up until the HIE 1.1 release. The first problem started appearing with with Stackage deprecating legacy snapshots, because the no-longer-maintained stack2nix tool, which was used to update all-hies, relied on them. As such, all-hies became frozen at version 1.1, leading to #52.

As time advanced, GHC 8.8.3 was released, and people started updating to it, leading to #55. So, we can't use stack2nix anymore, what to do? I thought about it and experimented with different approaches.

• We could use the Haskell tooling in nixpkgs itself, but this would be rather painful, require a lot of overriding, finding working nixpkgs versions, and hope that GHC doesn't need to be recompiled. Just overall messy and brittle
• We could sidestep nixpkgs and use hackage2nix directly, which even has some support for using stackage snapshots. However there's no easy support for the extra-deps field in HIE's stack.yaml, meaning this would have to be done manually, or somebody has to write a tool to convert that automatically, which is essentially what stack2nix did previously. This would be a whole lot of work and maintenance which I don't have the motivation for
• We could use the haskell.nix tooling, which is completely different than what nixpkgs uses, but its future looks very bright, even if it's a bit hard to understand currently (docs are being worked on though!)

So I went to try haskell.nix, which lead to #19 being picked up again. After some PRs to haskell.nix and a bunch of debugging, I was able to at least build HIE (version 1.4!). However, only GHC 8.6.5 and 8.8.3 work for now. This is supposed to be all-hies, but I guess let's just go with it, we can fix the other builds later.

Okay, I got working HIE builds again, let's try them out! I open a GHC 8.8.3 project of mine, only to discover that cabal-helper fails to do its thing. Turns out the cabal-helper version in HIE 1.4 just doesn't work with those dependencies, pretty sure it's a bug. I intended all-hies to mainly be stable versions, but let's roll with unstable for now where this is fixed.

So now I have unstable HIE building for GHC 8.6.5 and 8.8.3. I decide to try out a 8.6.5 project, only to again discover that cabal-helper fails to do its thing. Turns out it's pretty tricky to get a working cabal-install and Cabal version into the environment, which is why #58 is an issue.

Now let's also not ignore #32, which happens whenever your GHC's glibc version doesn't match the one all-hies uses, which is just annoying.

So, there are a bunch of problems, it's a pretty messy situation, nothing really works.

The plan for the future

It can't continue like this, we need some big changes for a smooth IDE experience. I have the following plans (motivated by above):

1. Only provide unstable HIE versions, so we can have fast fixes and updates if needed
2. Only provide GHC 8.6.5 and 8.8.3 (and other future main versions), these are the versions most people want to use, updating to those isn't very hard.
3. Use haskell.nix for the builds, this lets us automate the updates as best as possible without problems in the future
4. Provide cached builds for both glibc 2.27 and 2.30, these are the only recent versions available in nixpkgs (in nixos-19.09 and nixos-20.03 channels respectively). Also, IOHK provides cached GHC versions for those glibc versions.
5. all-hies will only expose a nixpkgs overlay in the future, this allows us to choose the correct glibc version, no matter what nixpkgs people use.
6. all-hies should newly be used per-project, no more global installation. A global installation will never work well with the glibc issue, and this plays well with it newly being an overlay
7. Provide a template project repository, which shows how to get a working Haskell development environment. This repository will pin nixpkgs and all-hies, use haskell.nix to provide the environment, expose a matching hoogle database for your project and provide a working cabal-install version.

These changes are not compatible with how all-hies currently works, meaning people will have to get an error about this. I believe all those changes will make an overall much better experience, as they should solve most of the current problems with all-hies.

I have a pretty good idea for how all those steps are done, so it shouldn't take too long for this to advance.

Ping @Anton-Latukha @mikelpr @bhipple

[mikelpr]

I didn't know people were actually globally installing! I've been using a hie.nix as follows:

{compiler? "ghc882"
,pkgs? (import ./nixpkgs.nix {compiler=compiler;})}:

let inherit (pkgs.haskell.lib) overrideCabal;
    inherit (pkgs.haskell.packages) ghc;
    all-hies = import (fetchTarball "https://github.com/infinisil/all-hies/tarball/master") {};
    
    drv = overrideCabal (ghc.callCabal2nix "dmt" ./. {})
    (old: {
      prePatch = "hpack";
      postPatch = "hoogle data all";
      buildTools = [
        ghc.cabal-install
        ghc.Cabal_3_0_0_0
        ghc.hoogle
        all-hies.versions.${compiler}
      ];
      # TODO hoogle hook
    });
in
  if pkgs.lib.inNixShell then drv.env else drv

(the hoogle and hpack patches don't actually work, I suppose they happen in a nix dir and that's not where hie takes as source directory nor would it be good using the intermediate nix dir as you'll be modifying the source on the project dir and not the nix build dir, but it'd be nice to find a way to automate these inside the nix expression)

this'd be the nixpkgs.nix I used on that project I'm sampling from:

{pkgs? (
  builtins.fetchGit {
    # Descriptive name to make the store path easier to identify
    #name = "nixos-unstable-2018-09-12";
    url = "https://github.com/nixos/nixpkgs-channels/";
    # Commit hash for nixos-unstable as of 2018-09-12
    # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-unstable`
    ref = "refs/heads/nixos-unstable";
    rev = "6bba9428d999c29832108622ef24efccc7f28632";
    #ref = "refs/heads/nixos-20.03";
    #rev = "5adf2a6c11646898742b0c08f7e94101620ba707";
  }
), compiler? "ghc882"}:

let config = {
  packageOverrides = pkgs: rec {
      haskell = pkgs.haskell // {
        packages = pkgs.haskell.packages // {
          ghc = pkgs.haskell.packages.${compiler}.override {
            overrides = self: super: with pkgs.haskell.lib; {
              persistent-postgresql = dontCheck super.persistent-postgresql;
              postgresql-simple = dontCheck super.postgresql-simple;
            };
          };
        };
      };
    };
  };
in (import pkgs {inherit config;})

I had to skip the postgresql tests there because they rely on a running pgsql server and even though it was running it just wouldn't connect when running the test. before I added postgresql into the mix I didn't need it as complicated but it's good to know how to modify whether to run tests on packages (and how is that done with haskell.nix)

and a nix-hie-wrapper like this:

if [[ -f 'hie.nix' ]];
  then
    nix-shell 'hie.nix' --pure --run "hpack" 1>/dev/null
    nix-shell 'hie.nix' --pure --run "hie-wrapper $@"
elif [[ -f 'default.nix' || -f 'shell.nix' ]]
  then nix-shell --pure --run "hie-wrapper $@"
else exec hie-wrapper "$@"
fi

that I set VSCode to use

you can base on them (and fix) for the template project repo. and adapt to haskell.nix. I'm excited at haskell.nix and would like to start using it actually

[sheyll]

Hey thank you so much for your work on this.

If you have time, could you tell my why hie does not include ghc?

IIUC, hie contains and depends on all kinds of tools and libraries among them ghc.

Something really great about Nix seems to me, to be able to have more or less isolated, reproducable and self-sufficient installations, by using full transitive closures containing baked-in nix-store paths to the dependencies.

So, while I might totally be Danning-Kruger-ing this, I wonder if it would be possible to have the IDE tooling come autonomous, batteries included, i.e. hie including a/the suitable ghc.

Maybe the implications could be, that the compilation outputs of a user submitted cabal build and an internal hie-triggered compilation must be separated and cannot be shared among each other. But to me that would be okay, since I am used to that from using other IDEs such as Eclipse and IntelliJ, that bring their own compiler.

[infinisil]

@mikelpr Nice! The template repository could maybe contain one version using the nixpkgs tooling and another using the haskell.nix one.

@sheyll In the Nix world, GHC binaries come preconfigured with the packages of your project, so e.g. you'd use pkgs.haskellPackages.ghcWithPackages (p: [ p.foo ]) to get a GHC that knows about the foo package. So while it would be possible for all-hies to ship with a GHC, it wouldn't know about all the packages in your project. A good idea however would be to write a wrapper around all-hies and some haskell tooling that sets everything up, including GHC, cabal-install, etc. This is what the template repository would do.

[poscat0x04]

I also made a repository for hie by reusing the code from hls-nix: https://github.com/poscat0x04/hie-nix.
It works gorgeously thanks to haskell.nix

[Anton-Latukha]

So you can split the effort and responsibility.

I never understand why people go the rivalry way when it is much more effective to discuss how to work together and split the effort and responsibility, use each-others gifted work and build your lean solutions and extend modules that provide features. The NIH syndrome is a staple of systematic issues and not growing. And reuse of work of others is the actual staple of quality of a person and the creation.

[bqv]

@Anton-Latukha I'm confused, is that a complaint or a congratulation?

[Anton-Latukha]

A statement.

[Anton-Latukha]

"There are forces in nature called Love and Hate. The force of Love causes elements to be attracted to each other and to be built up into some particular form..., and the force of Hate causes the decomposition of things." Empedoklēs, c. 494 – c. 434 BC.

[infinisil]

I just opened #64 to implement the things here, check there for future updates

[infinisil]

This is now implemented with #64 :)