Distribute sources.nix file with package

[cprussin]

This PR just adds the sources.nix file to the niv package distribution so that it can be referenced instead of copied into projects. This was briefly discussed in #159, I'm not sure if you really wanted to go in this direction @nmattia but I figured I'd put in this PR to help out just in case.

[cprussin]

One problem I realized with this approach is that sources.nix is used to get to niv itself, I'm sure there's some good solution here but I don't have any off the top of my head yet.

[michaelpj]

I think this would be problematic in a lot of cases. Now you have a bootstrap problem: how do you get the niv package so you can get sources.nix? Especially since many people use niv to pin nixpkgs.

[nmattia]

I was thinking the same, we'd need to basically copy portions of sources.nix to the generated default.nix. Also I think it's already fairly easy to grab the sources.nix at the moment:

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in pkgs.runCommand "sources.nix" {}
  "cp ${sources.niv}/nix/sources.nix $out"

[cprussin]

Just kicking around other ideas, this works currently with no changes:

{ nivSrc ? fetchTarball "https://github.com/nmattia/niv/tarball/master"
}:

let
  niv = import nivSrc {};
  sources = import "${nivSrc}/nix/sources.nix" { sourcesFile = ./sources.json; };
  pkgs = import sources.nixpkgs {};
in

pkgs.mkShell {
  buildInputs = [
    niv.niv
    pkgs.git
  ];
}

If the plan is to move to fetchurl for shell.nix, why not use fetchTarball for niv itself?

[cprussin]

I put together a commit in my dotfiles to see how it would feel to consume niv via fetchTarball (and consume shell.nix from the tarball) instead of bootstrapping. It actually feels really nice, with the expected caveat that if you want to make the niv version itself truly idempotent, you have to use fetchTarball { url = ...; sha256 = ...; } which gets back into managing dependencies the old pre-niv way. Of course, if the niv versions were tagged in the repo, you could mostly solve the developer experience by doing fetchTarball "https://github.com/nmattia/niv/tarball/v..." (but I'd still have a preference for setting the sha256 in my own code).

The more I think about it, even if you just plan to distribute shell.nix via fetchurl, you're going to have the same problem--either consumers must track shas manually since niv hasn't been bootstrapped yet, or consumers will lose some amount of guarantee of reproducability.

[nmattia]

Here's another idea:

let
  nivSrc = fetchTarball
    https://github.com/nmattia/niv/tarball/6f03879b8a91fb0c33c7186da347b914deee3efc;
  sources = import "${nivSrc}/nix/sources.nix" {
    sourcesFile = ./sources.json;
  };
  niv = import nivSrc {};
in

niv // sources

For reasons I can't understand right now, the following doesn't work:

let
  sources = import
    (builtins.fetchurl
      https://raw.githubusercontent.com/nmattia/niv/6f038/nix/sources.nix)
    { sourcesFile = ./sources.json; };
  niv = import sources.niv {};
in

niv // sources

It looks like anything that doesn't come from fetchTarball (that means builtins.fetchurl, "${./localfile}", etc) doesn't apply the argument (the default ./nix/sources.json is used). I'll need to experiment a bit more, but looks like a bug.

[cprussin]

Ha! @nmattia that's funny, that's pretty much exactly identical to what I ended up landing on in the commit I mentioned: https://github.com/cprussin/dotfiles/blob/master/sources.nix is actually identical to your post, except that I put a sha256 on the fetchTarball.

[nmattia]

I know, I copied it from there :)

[cprussin]

For reasons I can't understand right now, the following doesn't work:

let
  sources = import
    (builtins.fetchurl
      https://raw.githubusercontent.com/nmattia/niv/6f038/nix/sources.nix)
    { sourcesFile = ./sources.json; };
  niv = import sources.niv {};
in

niv // sources

I think one of the problems with something like this (even if it did work) is that now users need to be careful to sync the url they use for sources.nix with the version of niv that's in sources.json.

[nmattia]

users need to be careful to sync the url they use for sources.nix with the version of niv that's in sources.json.

I'm not sure I understand! Can you clarify?

[cprussin]

users need to be careful to sync the url they use for sources.nix with the version of niv that's in sources.json.

I'm not sure I understand! Can you clarify?

I think the versions of sources.nix and niv itself are bound together no? For instance, if I were to upgrade niv but still point to an old sources.nix, then niv could generate a sources.json that the version of sources.nix that I'm using doesn't understand. Vice versa, if I were to point to a new sources.nix but not upgrade niv, then my old niv could be generating deprecated sources.json that my new sources.nix no longer supports. So you have to make sure that sources.nix comes from the same niv package as the niv version that's being used to generate sources.json.

[michaelpj]

FWIW, I'd probably prefer to just keep checking in sources.nix, so I hope we keep that option.

Re: getting it from the package. This requires you to build niv before you can import sources.nix i.e. it's full on IFD, which has all the usual IFD problems (e.g. niv getting GC'd and repeatedly rebuilt when you build your project).

Fetching sources.nix directly doesn't have this problem. I think it's probably a matter of taste whether people prefer fetching it with a hash or just checking it in.

[nmattia]

I think the versions of sources.nix and niv itself are bound together no?

Well not necessarily; the sources.json tries to be as backwards compatible as possible. If we end up with something like this (@michaelpj btw checking it in will always be an option):

let
  sources = import
    (builtins.fetchurl
      https://raw.githubusercontent.com/nmattia/niv/6f038/nix/sources.nix)
    { sourcesFile = ./sources.json; };
  pkgs = import sources.nixpkgs {};
in pkgs.hello

Then the idea is still that niv init will update it. It will grep for 6f038 and replace it with the latest commit.

[michaelpj]

Then the idea is still that niv init will update it. It will grep for 6f038 and replace it with the latest commit.

See, this makes me uncomfortable. Now you're replacing bits of a non-niv-managed .nix file, which is not necessarily trivial to do. Part of what's nice about the separate file is that niv can just overwrite it without having to worry about anything else.

[cprussin]

Agreed with @michaelpj, that seems spooky to me. I'd rather just stick with having sources.nix be generated than have some program muck with non-managed sources. But I'm also paranoid and willing to try it and be proved wrong :)

[cprussin]

I'm gonna close this PR since it seems obvious to me that this particular idea isn't worth pursuing further, I'm happy to be involved in future discussion/brainsorming/guinnea pig-ing if you find it helpful.