Doesn't watch paths through IFD + filterSource (unsure if proper diagnosis)

[infinisil]

After @dudebout solved my problem in #6 of it watching too many paths, I'm encountering the opposite problem: it doesn't watch all paths it should.

A simple example:

let
  pkgs = import <nixpkgs> {};
  deps = pkgs.stdenv.mkDerivation {
    name = "deps";
    src = builtins.filterSource (path: type: baseNameOf path == "pkg") ./.;
    installPhase = ''
      echo "pkgs: with pkgs; [ $(cat pkg) ]" > $out
    '';
  };
in pkgs.stdenv.mkDerivation {
  name = "test";
  nativeBuildInputs = import deps pkgs;
}

This uses import-from-derivation (IFD) by first building a derivation for knowing the dependencies, depending on a file ./pkg through builtins.filterSource (touch that file), then puts those dependencies in the build inputs. Lorri ends up not watching the file, as seen from the log:

[2019-04-01T01:01:22Z DEBUG lorri] Input options: Arguments { verbosity: 0, command: Watch }
Started
[2019-04-01T01:01:22Z DEBUG lorri::build_loop] original paths: 518
[2019-04-01T01:01:22Z DEBUG lorri::build_loop]   -> reduced to: 3
[2019-04-01T01:01:22Z DEBUG lorri::build_loop] named drvs: {
    "shell": "/nix/store/gc1303z31vq35sl044fxfk0z9pmdkg42-test.drv",
    "shell_gc_root": "/nix/store/vl1rb2fsk4f50p5d6h2qsqj8iin86b20-lorri-keep-env-hack-test.drv"
}
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/gc1303z31vq35sl044fxfk0z9pmdkg42-test.drv" to "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-attr-shell"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/vl1rb2fsk4f50p5d6h2qsqj8iin86b20-lorri-keep-env-hack-test.drv" to "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-attr-shell_gc_root"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Adding root from "/nix/store/s11p9yzvbiff29m1qr30kk7hpcgij5cl-lorri-keep-env-hack-test" to "/home/infinisil/.cache/lorri/lorri/gc_root/build-0"
[2019-04-01T01:01:22Z DEBUG lorri::roots] Connecting root from "/home/infinisil/.cache/lorri/lorri/gc_root/build-0" to "/nix/var/nix/gcroots/per-user/infinisil/1aaab9fec1be4998a934b77cbe8194b8-ff9d02677be05e0746001cc0ed1ec807-build-0"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/Test/lorri/shell.nix"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/Test/lorri"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/.config/nixpkgs/config.nix"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/.config/nixpkgs"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching path "/home/infinisil/.config/nixpkgs/overlays"
[2019-04-01T01:01:22Z DEBUG lorri::watch] Watching parent path "/home/infinisil/.config/nixpkgs"
Completed(
    BuildResults {
        drvs: {
            0: "/home/infinisil/.cache/lorri/lorri/gc_root/build-0"
        },
        named_drvs: {
            "shell": "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell",
            "shell_gc_root": "/home/infinisil/.cache/lorri/lorri/gc_root/attr-shell_gc_root"
        }
    }
)

Meaning I can change the contents of the file ./pkg, which would have an influence on nix-shell, but lorri doesn't budge. And because there's no way to either forcefully include a file or forcefully reload everything, the only way to have it change the environment is by restarting lorri watch (or changing a different file).

While this may sound rather theoretical, that's actually how almost all functions in nixpkgs for developing Haskell packages work, including developPackage, callHackage, callCabal2nix, and more.

I am encountering this with my project (after an adjustment). Namely it doesn't reload when I add a dependency to the .cabal file, which is something you do a lot in Haskell.

[Profpatsch]

I’ve seen this, too. I think we’ll need some additional setup for properly recognizing IFD paths. I had started a minimal repro somewhere, will push it once it’s ready.

[infinisil]

To get around such problems, I'd love to have a lorri reload for manual (or externally triggered) reloading.

[Profpatsch]

Here’s an old issue I just migrated that talks about this: #31

[dudebout]

I tried to get around this issue for developing a cabal-based project, but ran into issues. Documenting this here in the hope it saves someone some time.

Haskell package environment sucks in the source
I ran cabal2nix by hand but lorri watched my whole directory. This surprised me, since the environment derived from the shell.nix produced by cabal2nix --shell . > shell.nix should not contain the source: it should only contain the dependencies of the package under development. This turns out to be a known issue: NixOS/nixpkgs#51079.

Haskell package environment not directly overridable
Since cabal2nix produces src = ./., I thought my issue might be due to #58. To get around it, I overrode the source with the following default.nix:

with import <nixpkgs> {};

let
  drv = haskellPackages.callPackage (import ./package.nix) { };
  drv' = drv.overrideAttrs (attrs: rec {
      src = nix-gitignore.gitignoreSource [".gitignore" "*.nix" "*.cabal" ".envrc"] ./.;
  })
in
  drv'.env

created the simple shell.nix:

import ./default.nix

and generated package.nix with cabal2nix . > package.nix. But this did not solve my problem. lorri was still watching the whole directory. Digging in the code, I think the issue is with the use of shellFor to generate the haskell package derivations env attribute introduced in NixOS/nixpkgs@5067773e39a. To be able to use shellFor, the fix operator is used which prevents the derivation from being truly overridable: overriding the source takes effect in the derivation but not in its env. The following default.nix is not ideal but fixes the issue:

with import <nixpkgs> {};

let
  drv = haskellPackages.callPackage (import ./package.nix) { };
  drv' = drv.overrideAttrs (attrs: rec {
      src = nix-gitignore.gitignoreSource [".gitignore" "*.nix" "*.cabal" ".envrc"] ./.;
  });
in
  haskellPackages.shellFor { packages = _: [drv']; }

[anka-213]

You don't even need IFD to reproduce this issue. Just running the imported derivation with a filtered source is sufficient:

let
  pkgs = import <nixpkgs> {};
  deps = pkgs.stdenv.mkDerivation {
    name = "deps";
    src = builtins.filterSource (path: type: baseNameOf path == "example") ./.;
    installPhase = ''
      echo "pkgs: with pkgs; [ $(cat example) ]" > $out
    '';
  };
in deps

Running the code that lorri uses internally to gather sources to watch shows that no reference to example is emmitted

$ nix-build -vv --no-out-link --expr "import ./logged-evaluation.nix { src = ./just-filter.nix; }" 2>&1 | grep example
<no output>

I haven't tried this with the latest version on master yet, so the results may have changed.

---

On a slightly unrelated note, callCabal2nix is supposed to have a filter already (see source code), but that seems to be kind of broken and the full source is included in the build for the cabal2nix-derivation anyways. :/

[dudebout]

@anka-213, in your example, lorri is not expected to track the file example. It is a source file that does not have an influence on the shell environment that nix-shell should set up. @infinisil's example is different because he uses deps in the nativeBuildInputs field, which means, the content of the pkg file will impact the content of the shell environment.

[anka-213]

@dudebout Aha, I see. However, the fact that nothing is logged in my example is probably the cause for not being able to track the sources through IFD. No info about what sources are being copied is being logged.

---

I am also a bit confused why manual filtering makes any difference when using callCabal2nix, since that already does filtering. For example:

pkgs.haskellPackages.callCabal2nix "example" ./. {}

will track the whole directory and nix-shell -vv will print

copied source '/path/to/example' -> '/nix/store/someHash-example'

while

pkgs.haskellPackages.callCabal2nix "example" (pkgs.lib.cleanSource ./.) {}

will not print such a copy statement and not track anything but the nix-file itself,
despite both commands copying exactly the same subset of the directory (the cabal-file) before generating the nix-derivation with cabal2nix.

Is this caused by NixOS/nixpkgs#51079? If so, adding an empty filter around src seems to be an extremely hacky workaround to that issue.

[dudebout]

I think what your are after is point 2. in #6 (comment)

[curiousleo]

@dudebout Aha, I see. However, the fact that nothing is logged in my example is probably the cause for not being able to track the sources through IFD. No info about what sources are being copied is being logged.

I have also come across this. Looking at Nix' source code, I got as far as follows:

• "copied source" is only logged by copyPathToStore
• copyPathToStore is only called from coerceToString
• coerceToString is called from all over the place

But it looks like the following are true:

• src = ./. or similar lead to copyPathToStore being called, so we get "copied source"
• src = builtins.filterSource (...) ./. and src = builtins.path (...) ./. use different code paths and no "copied source" is logged by Nix.

As a result, lorri adds the former to the watch list, but not the latter.

We had discussions about changing Nix' output some time ago, I think it is time to revive those discussions.

[curiousleo]

(Sorry about the spam, I just need a place to record this somewhere ...)

Grepping through the Nix codebase, here's what I've found: the path and filterSource primops both instantiate a PathFilter which is then passed around.

• rg -F PathFilter shows that the argument of type PathFilter is always called filter.
• rg '\Wfilter\(' shows that the only time a function called filter is called in the entire codebase is in archive.cc::dump.

So in order to capture which files, post-filter, Nix actually uses, one would need to record those files here (or somewhere downstream, e.g. in the consumer of the sink into which the filtered files are dumped).