From 17e859e53bfb374c514bc5d9762c902422a511bc Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:11:56 +0100 Subject: [PATCH 01/25] more informative title --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 0d8031c1b..788c6ce7d 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -1,5 +1,5 @@ (file-sets)= -# File sets +# Working with local files To build a local project in a Nix derivation, its source files must be accessible to the builder. From 285fd17c1d1c63b9adc51ba37565e4f11093ade5 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:13:14 +0100 Subject: [PATCH 02/25] shorten introduction - link to reference documentation on the `builder` executable - since the sandbox is enabled by default, we can just say that --- source/tutorials/file-sets.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 788c6ce7d..576b568ed 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -2,15 +2,9 @@ # Working with local files -To build a local project in a Nix derivation, its source files must be accessible to the builder. -But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled), -it won't have access to the local project files by default. - -To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, -whose paths are then accessible to derivation builders [^1]. - -[^1]: Technically only Nix store paths from the derivations inputs can be accessed, -but in practice this distinction is not important. +To build a local project in a Nix derivation, its source files must be accessible to the [`builder` executable](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). +Since by default the `builder` runs in an isolated environment that only allows reading from the Nix store, +the Nix language has built-in features to copy local files to the store and expose the resulting store paths. Using these features directly can be tricky however: From 6a61be98111f60476e46ff48d0788d50e5b74fb1 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:15:08 +0100 Subject: [PATCH 03/25] clarify wording --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 576b568ed..8f864fa03 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -15,7 +15,7 @@ Using these features directly can be tricky however: - The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) can address these problems. - However, it's hard to get the desired path selection using the `filter` function interface. + However, it's often hard to express the desired path selection using the `filter` function interface. In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. It abstracts over these functions with essentially the same functionality, From 7e7c6d79bb7c435889122a67b75a1e4bed07df9f Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:15:48 +0100 Subject: [PATCH 04/25] reword learning objective - sharpen the value proposition of the library - this meshes with and expands on the new title --- source/tutorials/file-sets.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 8f864fa03..fec4dfe98 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -17,9 +17,8 @@ Using these features directly can be tricky however: can address these problems. However, it's often hard to express the desired path selection using the `filter` function interface. -In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. -It abstracts over these functions with essentially the same functionality, -but an easier and safer interface. +In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. +It abstracts over built-in functionality and offers a safer and more convenient interface. ## Basics From 8402ecf299b4a0ee325c4a9547ee6f544c3d0675 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:17:18 +0100 Subject: [PATCH 05/25] Basics -> File sets since the title is now different, we can be more specific on what the first section is about: file sets, generally. --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index fec4dfe98..268b38e07 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -20,7 +20,7 @@ Using these features directly can be tricky however: In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. It abstracts over built-in functionality and offers a safer and more convenient interface. -## Basics +## File sets The file set library is based on the concept of _file sets_, a data type representing a collection of local files. From 06caaae956d268ac96fd078711d31d6ccaeaf1cf Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:18:17 +0100 Subject: [PATCH 06/25] clarify wording --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 268b38e07..e9ef4a5bd 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -24,7 +24,7 @@ It abstracts over built-in functionality and offers a safer and more convenient The file set library is based on the concept of _file sets_, a data type representing a collection of local files. -File sets can be created, composed and used with the various functions of the library. +File sets can be created, composed, and manipulated with the various functions of the library. The easiest way to experiment with the library is to use it through `nix repl`. From d388d6d633c1b556a87e3581f25570df9bc69993 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:18:34 +0100 Subject: [PATCH 07/25] add link to command reference --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index e9ef4a5bd..af42e4328 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -26,7 +26,7 @@ The file set library is based on the concept of _file sets_, a data type representing a collection of local files. File sets can be created, composed, and manipulated with the various functions of the library. -The easiest way to experiment with the library is to use it through `nix repl`. +The easiest way to experiment with the library is to use it through [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): ```shell-session $ nix repl -f channel:nixos-unstable From 76905abea74b49efa6f72a6a4ab17358cb614158 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:19:20 +0100 Subject: [PATCH 08/25] shorten introduction to `trace` --- source/tutorials/file-sets.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index af42e4328..662199757 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -34,9 +34,7 @@ $ nix repl -f channel:nixos-unstable nix-repl> fs = lib.fileset ``` -It's probably the easiest to just jump right in -by using the [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function, -which pretty-prints the files included in a given file set: +The [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function pretty-prints the files included in a given file set: ```shell-session nix-repl> fs.trace ./. null From 9fd9700c883a7f75be2acb27a1c989ae791661f8 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:20:24 +0100 Subject: [PATCH 09/25] use log ellipsis as in the other tutorials brackets in console output invoke weird associations, as they could be syntactically relevant. bare ellipses are unusual enough one can expect them to be intuitively parsed as artificial. --- source/tutorials/file-sets.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 662199757..2bd12c2bc 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -30,7 +30,7 @@ The easiest way to experiment with the library is to use it through [`nix repl`] ```shell-session $ nix repl -f channel:nixos-unstable -[...] +... nix-repl> fs = lib.fileset ``` @@ -231,7 +231,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/d8mj3z49s24q46rncma6v9kvi6xbx4vq-filesets.drv -[...] +... /nix/store/v5sx60xd9lvylgqcyqpchac1a2k8300c-filesets ``` @@ -242,7 +242,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/1xb412x3fzavr8d8c3hbl3fv9kyvj77c-filesets.drv -[...] +... /nix/store/n9i6gaf80hvbfplsv7zilkbfrz47s4kn-filesets ``` @@ -276,11 +276,11 @@ trace: - package.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/7960rh64d4zlkspmf4h51g4zys3lcjyj-filesets.drv -[...] +... /nix/store/aicvbzjvqzn06nbgpbrwqi47rxqdiqv9-filesets $ nix-build -[...] +... /nix/store/aicvbzjvqzn06nbgpbrwqi47rxqdiqv9-filesets ``` @@ -314,7 +314,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/ygpx17kshzc6bj3c71xlda8szw6qi1sr-filesets.drv -[...] +... /nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets $ nix-build @@ -343,7 +343,7 @@ trace: - package.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/zmgpqlpfz2jq0w9rdacsnpx8ni4n77cn-filesets.drv -[...] +... /nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets ``` @@ -441,7 +441,7 @@ trace: - string.txt (regular) this derivation will be built: /nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv building '/nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv'... -[...] +... /nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets $ touch src/select.o @@ -500,7 +500,7 @@ trace: - select.h (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/vn21azx8y06cjq80lrvib8ia4xxpwn3d-filesets.drv -[...] +... /nix/store/4xdfxm910x1i2qapv49caiibymfjhvla-filesets ``` From 5ca3739c36eb895e9b007acb90cbebb67ad4442d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:23:05 +0100 Subject: [PATCH 10/25] shorten introduction to path arguments --- source/tutorials/file-sets.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 2bd12c2bc..dc3984532 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -42,12 +42,9 @@ trace: /home/user (all files in directory) null ``` -You might wonder where the file set here is, because we just passed a [_path_](https://nixos.org/manual/nix/stable/language/values#type-path) to the function! - -The key is that for all functions that expect a file set for an argument, they _also_ accepts paths. -Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion). -The resulting file sets contain _all_ files under the given path. -We can see this from the trace `/home/user (all files in directory)` +All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path). +Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion), and the resulting file sets contain _all_ files under the given path. +In the previous trace this is indicated by `(all files in directory)`. Even though file sets conceptually contain local files, they _never_ add these files to the Nix store unless explicitly requested. From 568925ec8f01e237beb485f5fbfacd7b84ad5cd6 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:26:38 +0100 Subject: [PATCH 11/25] move tip before the other notes three highlight boxes in a row seem a bit too much, also the tip is relevant immediately after the first example. --- source/tutorials/file-sets.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index dc3984532..26dbbc48d 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -46,6 +46,17 @@ All functions that expect a file set for an argument also accept a [path](https: Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion), and the resulting file sets contain _all_ files under the given path. In the previous trace this is indicated by `(all files in directory)`. +:::{tip} +The `trace` function pretty-prints its first agument and returns its second argument. +But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: + +```shell-session +nix-repl> fs.trace ./. +trace: /home/user (all files in directory) +«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» +``` +::: + Even though file sets conceptually contain local files, they _never_ add these files to the Nix store unless explicitly requested. So even though we pretty-printed all files in your home directory, none of its contained files were imported because of that. @@ -63,17 +74,6 @@ the local files always get copied into the Nix store unless you use it within a Git repository! ::: -:::{tip} -The `trace` function pretty-prints its first agument and returns its second argument. -But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: - -```shell-session -nix-repl> fs.trace ./. -trace: /home/user (all files in directory) -«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» -``` -::: - This implicit coercion also works for files: ```shell-session From 97bbe4fcf7afd3cd2c98339acded1f60c4615301 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:31:40 +0100 Subject: [PATCH 12/25] restructure notes on copying files --- source/tutorials/file-sets.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 26dbbc48d..bc5ca0e25 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -57,15 +57,15 @@ trace: /home/user (all files in directory) ``` ::: -Even though file sets conceptually contain local files, -they _never_ add these files to the Nix store unless explicitly requested. -So even though we pretty-printed all files in your home directory, none of its contained files were imported because of that. +Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. +You don't have to worry about accidentally copying secrets into the world-readable store. + +In this example, although we pretty-printed the home directory, no files were copied. This is also evident from the expression evaluating instantly. -So you don't have to worry about accidentally copying secrets into the store. :::{note} -This is in contrast to coercion of paths to strings like in `"${./.}"`, -which _does_ add all of its contained files to the Nix store! +This is in contrast to coercion of paths to strings such as in `"${./.}"`, +which copies the whole directory to the Nix store on evaluation! ::: :::{warning} From fb6f6302d04c709d978abb282babc6b402cb573d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:32:36 +0100 Subject: [PATCH 13/25] add link to flakes documentation, add emphasis --- source/tutorials/file-sets.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index bc5ca0e25..5c359aad5 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -69,9 +69,7 @@ which copies the whole directory to the Nix store on evaluation! ::: :::{warning} -With current experimental Flakes, -the local files always get copied into the Nix store -unless you use it within a Git repository! +With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: From 767155bc15455637bd65bff2a87d092a9ac633fa Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:33:05 +0100 Subject: [PATCH 14/25] trim down phrasing --- source/tutorials/file-sets.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 5c359aad5..964c4cd44 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -80,11 +80,9 @@ trace: /etc/nix trace: - nix.conf (symlink) ``` -We can see that in addition to the included file, -it also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). +In addition to the included file, this also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). -But if we make a typo for a path that doesn't exist, -the library adequately complains about it: +If a given path doesn't exist, the library will complain: ```shell-session nix-repl> fs.trace /etc/nix/nix.nix From 152953ef302a6fd58bb5096c1b6ceccce9dd454b Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:34:52 +0100 Subject: [PATCH 15/25] A local directory -> Example project - more telling section title - use imperative in instructions --- source/tutorials/file-sets.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 964c4cd44..c0374de99 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -90,10 +90,10 @@ error: lib.fileset.trace: Argument (/etc/nix/nix.nix) is a path that does not exist. ``` -## A local directory +## Example project -To further experiment with the library, let's set up a local directory. -To start out, create a new directory, enter it, +To further experiment with the library, make a sample project. +Create a new directory, enter it, and set up `niv` to pin the Nixpkgs dependency: ```shell-session From 51a78a0bd4cc8c8f9126cf8e7ae54a8ecbdd5428 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:36:32 +0100 Subject: [PATCH 16/25] channel -> channel branch this follows the convention established in the FAQ on releases --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index c0374de99..4abdc99db 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -103,7 +103,7 @@ $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixo ``` :::{note} -For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. +We're using the `nixos-unstable` channel branch here, since no stable release has all the features needed for this tutorial. ::: Then create a `default.nix` file: From a6e5f2675f59c762fb78cb26aa98ccf78bd75dd8 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:38:01 +0100 Subject: [PATCH 17/25] package.nix -> build.nix this follows the convention established in other tutorials. also arguably, this file is *not* the package per se, but simply contains a representation of build instructions. --- source/tutorials/file-sets.md | 42 +++++++++++++++++------------------ 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 4abdc99db..9065a84c0 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -121,15 +121,15 @@ let inherit system; }; in -pkgs.callPackage ./package.nix { } +pkgs.callPackage ./build.nix { } ``` -From now on we'll just change the contents of `package.nix` while keeping `default.nix` the same. +From now on we'll just change the contents of `build.nix` while keeping `default.nix` the same. -For now, let's have a simple `package.nix` to verify everything works so far: +For now, let's have a simple `build.nix` to verify everything works so far: ```{code-block} nix -:caption: package.nix +:caption: build.nix { runCommand }: runCommand "hello" { } '' echo hello world @@ -160,10 +160,10 @@ That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-li It allows us to create a Nix store path containing exactly only the files that are in the file set, added to the Nix store rooted at a specific path. -Let's try it out by defining `package.nix` as follows: +Let's try it out by defining `build.nix` as follows: ```{code-block} nix -:caption: package.nix +:caption: build.nix { stdenv, lib }: let fs = lib.fileset; @@ -197,7 +197,7 @@ let's first use the file set library to include all files from the local directo and have it succeed by coping the `string.txt` file to the output: ```{code-block} nix -:caption: package.nix +:caption: build.nix :emphasize-lines: 4, 13-15 { stdenv, lib }: let @@ -254,7 +254,7 @@ that aren't in the second argument. Let's use it to filter out `./result` by changing the `sourceFiles` definition: ```{code-block} nix -:caption: package.nix +:caption: build.nix sourceFiles = fs.difference ./. ./result; ``` @@ -265,7 +265,7 @@ $ nix-build trace: /home/user/select trace: - default.nix (regular) trace: - nix (all files in directory) -trace: - package.nix (regular) +trace: - build.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/7960rh64d4zlkspmf4h51g4zys3lcjyj-filesets.drv @@ -296,7 +296,7 @@ we should use `maybeMissing` , -so let's try it: +Follow the instructions in the error message, and use `maybeMissing` to create a file set from a path that may not exist (in which case the file set will be empty): -```{code-block} nix + + +```{code-block} diff :caption: build.nix - sourceFiles = fs.difference ./. (fs.maybeMissing ./result); + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.difference ./. ./result; ++ sourceFiles = fs.difference ./. (fs.maybeMissing ./result); + in ``` -This now works, reliably filtering out `./result` if it exists: +This now works, using the whole directory since `./result` is not present: ``` $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: - /nix/store/ygpx17kshzc6bj3c71xlda8szw6qi1sr-filesets.drv + /nix/store/zr19bv51085zz005yk7pw4s9sglmafvn-fileset.drv ... -/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets +/nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset +``` + +Another build attempt will produce a different trace, but the same output path: +``` $ nix-build -trace: /home/user/select +trace: /home/user/fileset +trace: - build.nix (regular) trace: - default.nix (regular) +trace: - hello.txt (regular) trace: - nix (all files in directory) -trace: - build.nix (regular) -trace: - string.txt (regular) -/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets +trace: - world.txt (regular) +/nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset ``` -## Adding file sets together +## Union (exclude) + +There is still a problem: +Changing _any_ of the included files causes the derivation to be rebuilt, even though it doesn't depend on those files. -We still have a problem however: -Changing _any_ of the included files causes the derivation to be rebuilt, -even though it doesn't depend on those files. +Append an empty line to `build.nix`: ```shell-session -$ echo "# Just a comment" >> build.nix +$ echo >> build.nix +``` + +Again, Nix will start from scratch: +```shell-session $ nix-build trace: /home/user/select trace: - default.nix (regular) @@ -346,77 +439,104 @@ this derivation will be built: /nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets ``` -One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions) -to create a file set containing all of the files we don't want, -and removing that instead: +One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions). -```{code-block} nix +Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`): + +```{code-block} diff :caption: build.nix - sourceFiles = - fs.difference - ./. - (fs.unions [ - (fs.maybeMissing ./result) - ./default.nix - ./build.nix - ./nix - ]); + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.difference ./. (fs.maybeMissing ./result); ++ sourceFiles = ++ fs.difference ++ ./. ++ (fs.unions [ ++ (fs.maybeMissing ./result) ++ ./default.nix ++ ./build.nix ++ ./nix ++ ]); + in ``` -This also gives us the opportunity to show the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, -which as the name implies, allows filtering the files in a local path. -We use it to select all files whose name ends with `.nix`: +Changing any of the excluded files now doesn't necessarily require a rebuild anymore. -```{code-block} nix -:caption: build.nix - sourceFiles = - fs.difference - ./. - (fs.unions [ - (fs.maybeMissing ./result) - (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) - ./nix - ]); -``` +Check it and modify one of the excluded files again: -Changing any of the removed files now doesn't necessarily require a rebuild anymore: +``` +$ echo >> build.nix +``` -```shell-session +``` $ nix-build -trace: /home/user/select -trace: - string.txt (regular) -/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets +trace: /home/user/fileset +trace: - hello.txt (regular) +trace: - world.txt (regular) +/nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset +``` + +## Filter + +Dealing with a large number of files, independent of their location, can be done programmatically with the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function. -$ echo "# Just a comment" >> build.nix +Use it to select all files with a name ending in `.nix`: +```{code-block} diff +:caption: build.nix + { stdenv, lib }: + let + fs = lib.fileset; + sourceFiles = + fs.difference + ./. + (fs.unions [ + (fs.maybeMissing ./result) +- ./default.nix +- ./build.nix ++ (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) + ./nix + ]); + in +``` + +This does not change the result: + +```shell-session $ nix-build -trace: /home/user/select -trace: - string.txt (regular) -/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets +trace: /home/user/fileset +trace: - hello.txt (regular) +trace: - world.txt (regular) +/nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset ``` -Notable with this approach is that new files added to the current directory are _included_ by default. +Notably, with this approach, new files added to the source directory are _included_ by default. Depending on your project, this might be a better fit than the alternative in the next section. -## Only including necessary files +## Union (include) -To contrast the above approach, we can also directly use `unions` to select only the files we want to _include_. -This means that new files added to the current directory would be _excluded_ by default. +To contrast the previous approach, `unions` can also be used to select only the files to _include_. +This means that new files added to the current directory would be ignored by default. -To demonstrate, let's create some extra files to select: +Create some additional files: ```shell-session $ mkdir src $ touch build.sh src/select.{c,h} ``` -And then create a file set from just the ones we're interested in: +Then create a file set from only the files to be included explicitly: ```{code-block} nix :caption: build.nix +{ stdenv, lib }: +let + fs = lib.fileset; sourceFiles = fs.unions [ + ./hello.txt + ./world.txt ./build.sh - ./string.txt (fs.fileFilter (file: lib.hasSuffix ".c" file.name @@ -425,49 +545,72 @@ And then create a file set from just the ones we're interested in: ./src ) ]; +in + +fs.trace sourceFiles + +stdenv.mkDerivation { + name = "fileset"; + src = fs.toSource { + root = ./.; + fileset = sourceFiles; + }; + postInstall = '' + cp -vr $src $out + ''; +} ``` -When building this you'll see that only the specified files are used, even when a new one is added: +The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately: ```shell-session $ nix-build -trace: /home/user/select +trace: /home/user/fileset trace: - build.sh (regular) -trace: - src -trace: - select.c (regular) -trace: - select.h (regular) -trace: - string.txt (regular) +trace: - hello.txt (regular) +trace: - src (all files in directory) +trace: - world.txt (regular) this derivation will be built: - /nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv -building '/nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv'... + /nix/store/sjzkn07d6a4qfp60p6dc64pzvmmdafff-fileset.drv ... -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source' -> '/nix/store/zl4n1g6is4cmsqf02 +dci5b2h5zd0ia4r-fileset' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/build.sh' -> '/nix/store/zl4n1g6i +s4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/hello.txt' -> '/nix/store/zl4n1g6 +is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/world.txt' -> '/nix/store/zl4n1g6 +is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src' -> '/nix/store/zl4n1g6is4cms +qf02dci5b2h5zd0ia4r-fileset/src' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.c' -> '/nix/store/zl4n +1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.h' -> '/nix/store/zl4n +1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' +... +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset +``` +Only the specified files are used, even when a new one is added: + +```shell-session $ touch src/select.o $ nix-build -trace: /home/user/select trace: - build.sh (regular) +trace: - hello.txt (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +trace: - world.txt (regular) +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` -## Git +## Matching files tracked by Git -In case we track files with Git, we can use [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) to re-use the same set of files by Git. +If a directory is part of a Git repository, [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) automatically filters for files that are tracked by Git. -:::{note} -With current experimental Flakes, -it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, -even with `nix build path:.`. -However it's also not needed, because by default, -`nix build` only allows access to Git-tracked files. -::: - -Let's create a local Git repository and add track all files except `src/select.o` and `./result` to it: +Create a local Git repository and add all files except `src/select.o` and `./result` to it: ```shell-session $ git init @@ -488,28 +631,40 @@ Building we get ```shell-session $ nix-build warning: Git tree '/home/user/select' is dirty -trace: /home/user/select +trace: /home/vg/src/nix.dev/fileset +trace: - build.nix (regular) trace: - build.sh (regular) trace: - default.nix (regular) +trace: - hello.txt (regular) trace: - nix (all files in directory) -trace: - build.nix (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) +trace: - world.txt (regular) this derivation will be built: - /nix/store/vn21azx8y06cjq80lrvib8ia4xxpwn3d-filesets.drv + /nix/store/p9aw3fl5xcjbgg9yagykywvskzgrmk5y-fileset.drv ... -/nix/store/4xdfxm910x1i2qapv49caiibymfjhvla-filesets +/nix/store/cw4bza1r27iimzrdbfl4yn5xr36d6k5l-fileset ``` -This includes too much though, we don't need all of these files to build the derivation. +This includes too much though, as not all of these files are needed to build the derivation as originally intended. + +:::{note} +With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with + +```shell-session +nix build path:. +``` + +However it's also not needed, because by default, `nix build` only allows access to files tracked by Git. +::: ## Intersection This is where `intersection` comes in. -It allows us to create a file set consisting only of files that are in _both_ of two file sets. -In this case we only want files that are both tracked by git, and included in our exclusive selection: +It allows creating a file set that consists only of files that are in _both_ of the two given file sets. + +Select all files that are both tracked by Git *and* relevant for the build: ```{code-block} nix :caption: build.nix @@ -517,24 +672,25 @@ In this case we only want files that are both tracked by git, and included in ou fs.intersection (fs.gitTracked ./.) (fs.unions [ + ./hello.txt + ./world.txt ./build.sh - ./string.txt ./src ]); ``` -At last we get what we expect: +This will produce the same output as in the other approach and therefore re-use a previous build result: ```shell-session $ nix-build warning: Git tree '/home/user/select' is dirty -trace: /home/user/select trace: - build.sh (regular) +trace: - hello.txt (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +trace: - world.txt (regular) +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` ## Conclusion From fa037e69f6237c1d3a9fcce8bf39f818e445ce9e Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 05:19:27 +0100 Subject: [PATCH 22/25] fix omission Co-authored-by: Silvan Mosberger --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index f6a41f8dc..41753f58c 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -203,7 +203,7 @@ t.drv'. That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in: It adds exactly the files in the given file set to a directory in the Nix store, starting from a specified root path. -Define `package.nix` as follows: +Define `build.nix` as follows: ```{code-block} nix :caption: build.nix From 429bbb0621b8049b151446d2240b594d6cb3f203 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:24:47 +0100 Subject: [PATCH 23/25] Update source/tutorials/file-sets.md Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 41753f58c..63b2c1b99 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -413,7 +413,7 @@ trace: - world.txt (regular) /nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset ``` -## Union (exclude) +## Union (explicitly exclude files) There is still a problem: Changing _any_ of the included files causes the derivation to be rebuilt, even though it doesn't depend on those files. From 16db5641264c74e3a3b72bbd4c8fc940726674e2 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:25:51 +0100 Subject: [PATCH 24/25] Update source/tutorials/file-sets.md Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 63b2c1b99..520a342b6 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -511,7 +511,7 @@ trace: - world.txt (regular) /nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset ``` -Notably, with this approach, new files added to the source directory are _included_ by default. +Notably, the approach of using `difference ./.` explicitly selects the files to _exclude_, which means that new files added to the source directory are included by default. Depending on your project, this might be a better fit than the alternative in the next section. ## Union (include) From 993bbc17879438c14f90ca60ede792f835adbc3c Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:30:16 +0100 Subject: [PATCH 25/25] Apply suggestions from code review --- source/tutorials/file-sets.md | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 520a342b6..276c863f8 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -69,7 +69,7 @@ which copies the whole directory to the Nix store on evaluation! ::: :::{warning} -With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! +With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: @@ -514,7 +514,7 @@ trace: - world.txt (regular) Notably, the approach of using `difference ./.` explicitly selects the files to _exclude_, which means that new files added to the source directory are included by default. Depending on your project, this might be a better fit than the alternative in the next section. -## Union (include) +## Union (explicitly include files) To contrast the previous approach, `unions` can also be used to select only the files to _include_. This means that new files added to the current directory would be ignored by default. @@ -556,7 +556,7 @@ stdenv.mkDerivation { fileset = sourceFiles; }; postInstall = '' - cp -vr $src $out + cp -vr . $out ''; } ``` @@ -573,20 +573,13 @@ trace: - world.txt (regular) this derivation will be built: /nix/store/sjzkn07d6a4qfp60p6dc64pzvmmdafff-fileset.drv ... -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source' -> '/nix/store/zl4n1g6is4cmsqf02 -dci5b2h5zd0ia4r-fileset' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/build.sh' -> '/nix/store/zl4n1g6i -s4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/hello.txt' -> '/nix/store/zl4n1g6 -is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/world.txt' -> '/nix/store/zl4n1g6 -is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src' -> '/nix/store/zl4n1g6is4cms -qf02dci5b2h5zd0ia4r-fileset/src' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.c' -> '/nix/store/zl4n -1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.h' -> '/nix/store/zl4n -1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' +'.' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset' +'./build.sh' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' +'./hello.txt' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' +'./world.txt' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' +'./src' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src' +'./src/select.c' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' +'./src/select.h' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' ... /nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` @@ -650,7 +643,7 @@ this derivation will be built: This includes too much though, as not all of these files are needed to build the derivation as originally intended. :::{note} -With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with +With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with ```shell-session nix build path:.