Boot packages missing from Hackage cannot be used.

[TravisWhitaker]

TL;DR: How can haskell.nix correctly handle plans that depend on boot packages that haven't been uploaded to Hackage? One solution is to always copy the in-use GHC's boot packages from the global package database into the build package database, as is already done for non-reinstallable boot packages.

Packages depending on GHC boot packages that have not been uploaded to Hackage cannot be built with haskell.nix. By "boot packages," I mean precisely what the GHC docs say; the code in this project seems to play it fast-and-loose with the term "boot package", which has caused me hours of confusion. There is some existing machinery in haskell.nix (specifically this stuff) that does the right thing for a seemingly-arbitrary subset of boot packages like base and bytestring, but fails for packages like Cabal.

I've uploaded a test package here that demonstrates this issue by depending on base-4.13.0.0 and Cabal-3.0.1.0, boot packages that are not on Hackage. Attempting to build this package yields this:

$ nix repl
Welcome to Nix version 2.3.3. Type :? for help.

nix-repl> p = import <nixpkgs> (import ./.)

nix-repl> x = p.haskell-nix.cabalProject {name = "troll"; src = builtins.fetchGit {url = "https://github.com/TravisWhitaker/troll"; ref = "master"; rev = "d23202e5cce360859e0d449f0bdd1a46471ab9a2";}; ghc = p.buildPackages.haskell-nix.compiler.ghc883;}

nix-repl> :b x.troll.components.all
trace: Using latest index state for troll!
trace: [troll-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: Using index-state: 2020-03-27T00:00:00Z for troll
trace: [alex-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [happy-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [hscolour-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
error: attribute '3.0.1.0' missing, at /nix/store/9bzvw2f7fya8pcf9mn8r27i18iyn6wdb-troll-plan-to-nix-pkgs/default.nix:17:30

By examining the plan, we can see the issue:

nix-repl> x = p.haskell-nix.callCabalProjectToNix {name = "troll"; src = builtins.fetchGit {url = "https://github.com/TravisWhitaker/troll"; ref = "master"; rev = "d23202e5cce360859e0d449f0bdd1a46471ab9a2";};
ghc = p.buildPackages.haskell-nix.compiler.ghc883;}
nix-repl> :b x.projectNix
trace: Using latest index state for troll!
trace: [troll-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: Using index-state: 2020-03-27T00:00:00Z for troll

this derivation produced the following outputs:
  out -> /nix/store/9bzvw2f7fya8pcf9mn8r27i18iyn6wdb-troll-plan-to-nix-pkgs

Inside /nix/store/9bzvw2f7fya8pcf9mn8r27i18iyn6wdb-troll-plan-to-nix-pkgs/default.nix we find:

{
  pkgs = hackage:
    {
      packages = {
        "binary".revision = (((hackage."binary")."0.8.7.0").revisions).default;
        "ghc-prim".revision = (((hackage."ghc-prim")."0.5.3").revisions).default;
        "unix".revision = (((hackage."unix")."2.7.2.2").revisions).default;
        "mtl".revision = (((hackage."mtl")."2.2.2").revisions).default;
        "rts".revision = (((hackage."rts")."1.0").revisions).default;
        "deepseq".revision = (((hackage."deepseq")."1.4.4.0").revisions).default;
        "parsec".revision = (((hackage."parsec")."3.1.14.0").revisions).default;
        "directory".revision = (((hackage."directory")."1.3.6.0").revisions).default;
        "template-haskell".revision = (((hackage."template-haskell")."2.15.0.0").revisions).default;
        "containers".revision = (((hackage."containers")."0.6.2.1").revisions).default;
        "bytestring".revision = (((hackage."bytestring")."0.10.10.0").revisions).default;
        "text".revision = (((hackage."text")."1.2.4.0").revisions).default;
        "Cabal".revision = (((hackage."Cabal")."3.0.1.0").revisions).default;
        "base".revision = (((hackage."base")."4.13.0.0").revisions).default;
        "time".revision = (((hackage."time")."1.9.3").revisions).default;
        "transformers".revision = (((hackage."transformers")."0.5.6.2").revisions).default;
        "filepath".revision = (((hackage."filepath")."1.4.2.1").revisions).default;
        "process".revision = (((hackage."process")."1.6.8.0").revisions).default;
        "pretty".revision = (((hackage."pretty")."1.1.3.6").revisions).default;
        "ghc-boot-th".revision = (((hackage."ghc-boot-th")."8.8.3").revisions).default;
        "array".revision = (((hackage."array")."0.5.4.0").revisions).default;
        "integer-gmp".revision = (((hackage."integer-gmp")."1.0.2.0").revisions).default;
        };
      compiler = {
        version = "8.8.3";
        nix-name = "ghc883";
        packages = {
          "binary" = "0.8.7.0";
          "ghc-prim" = "0.5.3";
          "unix" = "2.7.2.2";
          "mtl" = "2.2.2";
          "rts" = "1.0";
          "deepseq" = "1.4.4.0";
          "parsec" = "3.1.14.0";
          "directory" = "1.3.6.0";
          "template-haskell" = "2.15.0.0";
          "containers" = "0.6.2.1";
          "bytestring" = "0.10.10.0";
          "text" = "1.2.4.0";
          "Cabal" = "3.0.1.0";
          "base" = "4.13.0.0";
          "time" = "1.9.3";
          "transformers" = "0.5.6.2";
          "filepath" = "1.4.2.1";
          "process" = "1.6.8.0";
          "pretty" = "1.1.3.6";
          "ghc-boot-th" = "8.8.3";
          "array" = "0.5.4.0";
          "integer-gmp" = "1.0.2.0";
          };
        };
      };
  extras = hackage:
    { packages = { troll = ./.plan.nix/troll.nix; }; };
  modules = [
    ({ lib, ... }:
      { packages = { "template-haskell-hello" = { flags = {}; }; }; })
    ];
  }

This plan has some strange things in it. We depend on packages like rts-1.0, Cabal-3.0.1.0, base-4.13.0.0, and ghc-boot-th-8.8.3 that are not on Hackage at all. However, we only get a complaint when depending on Cabal. My hunch is that this slips by because Cabal is somehow treated differently than the non-reinstallable boot packages.

excludeBootPackages is where the first part of this filtering occurs. In this case it will exclude from the plan everything in ghc-boot-pkgs.ghc883, which is:

nix-repl> builtins.attrNames p.ghc-boot-packages.ghc883
[ "base" "bytestring" "ghc" "ghc-boot" "ghc-heap" "ghci" "iserv" "iserv-proxy" "libiserv" "remote-iserv" ]

This is a strange subset of boot packages. ghc and bytestring are reinstallable boot packages, so it's not clear why they should be excluded; while some wired-in packages like ghc-prim and template-haskell are missing.

In the case of the troll test package, excludeBootPackages explains why we don't try and fail to download base-4.13.0.0 from Hackage, but what about the others? There's code here that strips config.bootPkgs from the plan, which comes from here. This gets rid of rts and ghc-boot-th, leaving Cabal as the only remaining missing package.

The component-driver also has a list of non-reinstallable boot packages, which haskell.nix knows to copy from the global package database. If I set config.reinstallableLibGhc = false (so that Cabal is included in config.nonReinstallablePkgs) and append config.nonReinstallablePkgs to config.bootPkgs, then I'm successfully able to build the troll package.

I think the right way for haskell.nix to handle all boot packages is this: always copy all potentially-non-reinstallable boot packages from the global package database. If a plan contains a Hackage reference to a newer version of a reinstallable boot package and we trust that the plan is consistent, then I think it should be fine for those multiple boot package versions to sit in the same package database. Always copying these packages will also fix #481, since we won't care if the Stackage snapshot contains the boot packages anymore.

@angerman @hamishmack curious to hear your thoughts. Is this line of reasoning correct? Are there better ways of addressing this?

[TravisWhitaker]

Now that Cabal-3.0.1.0 is on Hackage, one must go back to b4823ea2db0cd610bf658d6c6ccdcacf9ef47e56 or so to reproduce this.

[angerman]

@TravisWhitaker As this is not finally resolved upstream (hackage), are you ok with closing this?

[TravisWhitaker]

I suppose it depends on what the behavior of haskell.nix should be. Is it desirable to run plans that include boot packages that aren't on hackage? I think it'd be useful to handle this, since this issue of boot package releases missing from hackage has occurred the past several GHC releases, and is likely to continue in the future.

[angerman]

There is validation support in ghc’s build machinery to ensure packages are on hackage. The cabal issue recently was due to hackage not supporting cabal3 packages. Of course this should have been fixed prior to releasing ghc, but, ohh well...

Thus I hope the risk of having ghc ship with packages not on hackage going forward is minimal.

/cc @bgamari

[codygman]

Thus I hope the risk of having ghc ship with packages not on hackage going forward is minimal.

That's sadly already proved to not be true in haskell-infra/hackage-trustees#240 for GHC 8.10 and previous releases as catalogued there and I fear it'll be the same in the future. I

think haskell.nix is going to be uniquely negatively affected here by boot packages not being on hackage quickly.

I really don't understand the situation and you might take away from that thread as I did there isn't much transparency or interest in an exposition of the core issue so that others may be able to help. I sense there is an internal consensus of "this problem can't be helped so our time is more useful fixing other things than describing something that can't be fixed or automated". So can't say whether that's good or bad, reasonable or unreasonable, only there's not much transparency and it seems very difficult to learn enough alone to contribute meaningfully.

I've personally been running into issues from the linked issue for a while know, so sorry for some venting. My main point was haskell.nix might want to consider delayed boot packages a reality for hard to discover reasons and act accordingly given recent and past history.