Add positive source filter

[roberth]

Motivation

This improves the source filter as promised.

Source filtering is a really nice Nix feature that lets us avoid a lot of rebuilds, which speeds up the iteration cycle a lot in cases where the relevant source files aren't actually modified.

We used to have a source filter that marked a few files as irrelevant, but this is the wrong approach, as we have many more files that are irrelevant. We may call this negative filtering.

This commit switches the source filtering to positive filtering, which is a lot more robust. Instead of marking which files we don't need we marked the files that we do need.

It's a superior approach because it is fail safe. Instead of allowing build performance problems to creep in over time, we require that all source inputs are declared.

I shouldn't have to explain that declaring inputs is a good practice, so I'll stop over-explaining here.

I do have to acknowledge that this will cause a build failure when the filter is incomplete. This is good, because it's the only realistic way we could be reminded of these problems. These events will be infrequent, so the small cost of extending the filter is worth it, compared to the hidden cost of longer dev cycles for things like tests, docker image, etc, etc.

Context

• I encountered unnecessary rebuilds again in flake: update nixpkgs: 22.11 -> 23.05 #8569. It's a drag.

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• agreed on idea
• agreed on implementation strategy
• tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests - tests/nixos/*
• documentation in the manual
• documentation in the internal API docs
• code and comments are self-explanatory
• commit message explains why the change was made
• new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

[Ericson2314]

If this is from an unmerged PR is there a risk it would go away?

Could we merge it and back port it? Or is that too much?

[roberth]

I've added the tree to this repo using a merge commit now.

[Ericson2314]

Definitely support the concept. This is much better than before. Just the one question on where we are fetching the filtering library from.

[infinisil]

Really nice to see the file set library being used here! I do have to point out that the code from the PR you're using (NixOS/nixpkgs#222981) hasn't been properly tested, but the first mergeable part of it (NixOS/nixpkgs#245623) now has a great test suite :)

[Ericson2314]

@infinisil Are you saying that if we merge that second part, and backport it, it is enough for this?

[infinisil]

@Ericson2314 Nope, it will take a bunch more work to get all the parts necessary for this. That second PR only adds lib.fileset.toSource, this here uses lib.fileset.{union,intersect,difference,fileFilter} as well

[Ericson2314]

OK sounds good, thanks for approving this @infinisil and clarifying to me the Nixpkgs situation.

@roberth has also explained to me what is going on with the fetchTarball of the tree hash above. (I am delighted GitHub allows downloading by tree hash!) I want to add more detail to the comment, but we need to get to 23.05 to fix some things, so I will do that after we merge this.