From 7e962264089c0a42ae1692edfe64bb5f73617178 Mon Sep 17 00:00:00 2001 From: Jacek Galowicz Date: Sat, 31 Aug 2024 21:45:14 +0200 Subject: [PATCH] Add tutorial on setting up a binary cache (#1040) * Add tutorial about binary cache setup Co-authored-by: Valentin Gagarin Co-authored-by: wamirez --- source/guides/recipes/add-binary-cache.md | 4 + source/guides/recipes/post-build-hook.md | 1 + source/tutorials/nixos/binary-cache-setup.md | 237 ++++++++++++++++++ source/tutorials/nixos/index.md | 5 + .../nixos/provisioning-remote-machines.md | 6 + 5 files changed, 253 insertions(+) create mode 100644 source/tutorials/nixos/binary-cache-setup.md diff --git a/source/guides/recipes/add-binary-cache.md b/source/guides/recipes/add-binary-cache.md index e454c201d..e360b63b9 100755 --- a/source/guides/recipes/add-binary-cache.md +++ b/source/guides/recipes/add-binary-cache.md @@ -3,6 +3,10 @@ Nix can be configured to use a binary cache with the [`substituters`](https://nix.dev/manual/nix/2.21/command-ref/conf-file.html#conf-substituters) and [`trusted-public-keys`](https://nix.dev/manual/nix/2.21/command-ref/conf-file.html#conf-trusted-public-keys) settings, either exclusively or in addition to cache.nixos.org. +:::{tip} +Follow the tutorial to [set up an HTTP binary cache](setup-http-binary-cache) and create a key pair for signing store objects. +::: + For example, given a binary cache at `https://example.org` with public key `My56...Q==%`, and some derivation in `default.nix`, make Nix exclusively use that cache once by passing [settings as command line flags](https://nix.dev/manual/nix/2.21/command-ref/conf-file#command-line-flags): ```shell-session diff --git a/source/guides/recipes/post-build-hook.md b/source/guides/recipes/post-build-hook.md index 700240ffb..8b1525f59 100644 --- a/source/guides/recipes/post-build-hook.md +++ b/source/guides/recipes/post-build-hook.md @@ -1,3 +1,4 @@ +(post-build-hooks)= # Setting up post-build hooks This guide shows how to use the Nix [`post-build-hook`](https://nix.dev/manual/nix/2.22/command-ref/conf-file#conf-post-build-hook) configuration option to automatically upload build results to an [S3-compatible binary cache](https://nix.dev/manual/nix/2.22/store/types/s3-binary-cache-store). diff --git a/source/tutorials/nixos/binary-cache-setup.md b/source/tutorials/nixos/binary-cache-setup.md new file mode 100644 index 000000000..3372ce5d8 --- /dev/null +++ b/source/tutorials/nixos/binary-cache-setup.md @@ -0,0 +1,237 @@ +--- +myst: + html_meta: + "description lang=en": "Setting up a binary cache for store objects" + "keywords": "Nix, caching" +--- + +(setup-http-binary-cache)= +# Setting up an HTTP binary cache + +A binary cache stores pre-built [Nix store objects](https://nix.dev/manual/nix/latest/store/store-object) and provides them to other machines over the network. +Any machine with a Nix store can be an binary cache for other machines. + +## Introduction + +In this tutorial you will set up a Nix binary cache that will serve store objects from a NixOS machine over HTTP or HTTPS. + +### What will you learn? + +You'll learn how to: +- Set up signing keys for your cache +- Enable the right services on the NixOS machine serving the cache +- Check that the setup works as intended + +### What do you need? + +- A working [Nix installation]() on your local machine + +- SSH access to a NixOS machine to use as a cache + + If you're new to NixOS, learn about the [module system](module-system-deep-dive) and configure your first system with [](nixos-vms). + +- (optional) A public IP and DNS domain + + If you don't host yourself, check [NixOS friendly hosters](https://wiki.nixos.org/wiki/NixOS_friendly_hosters) on the NixOS Wiki. + Follow the tutorial on [](provisioning-remote-machines) to deploy your NixOS configuration. + +For a cache on a local network, we assume: +- The hostname is `cache` (replace it with yours, or an IP address) +- The host serves store objects via HTTP on port 80 (this is the default) + +For a publicly accessible cache, we assume: +- The domain name is `cache.example.com` (replace it with yours) +- The host serves store objects via HTTPS on port 443 (this is the default) + +### How long will it take? + +- 25 minutes + +## Set up services + +For the NixOS machine hosting the cache, create a new configuration module in `binary-cache.nix`: + +```{code-block} nix +{ config, ... }: + +{ + services.nix-serve = { + enable = true; + secretKeyFile = "/var/secrets/cache-private-key.pem"; + }; + + services.nginx = { + enable = true; + recommendedProxySettings = true; + virtualHosts.cache = { + locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}"; + }; + }; + + networking.firewall.allowedTCPPorts = [ + config.services.nginx.defaultHTTPListenPort + ]; +} +``` + +The options under [`services.nix-serve`] configure the binary cache service. + +`nix-serve` doesn't support IPv6 or SSL/HTTPS. +The [`services.nginx`] options are used to set up a proxy, which does support IPv6, to handle requests to the hostname `cache`. + +[`services.nix-serve`]: https://search.nixos.org/options?query=services.nix-serve +[`services.nginx`]: https://search.nixos.org/options?query=services.nginx + +:::{important} +There is an [optional HTTPS section](https-binary-cache) at the end of this tutorial. +::: + +Add the new NixOS module to the existing machine configuration: + +```{code-block} nix +{ config, ... }: + +{ + imports = [ + ./binary-cache.nix + ]; + + # ... +} +``` + +From your local machine, deploy the new configuration: + +```shell-session +nixos-rebuild switch --no-flake --target-host root@cache +``` + +:::{note} +The binary cache daemon will report errors because there is no secret key file, yet. +::: + +## Generate a signing key pair + +A pair of private and public keys is required to ensure that the store objects in the cache are authentic. + + + +To generate a key pair for the binary cache, replace the example hostname `cache.example.com` with your hostname: + +```shell-session +nix-store --generate-binary-cache-key cache.example.com cache-private-key.pem cache-public-key.pem +``` + +`cache-private-key.pem` will be used by the binary cache daemon to sign the binaries as they are served. +Copy it to the location configured in `services.nix-serve.secretKeyFile` on the machine hosting the cache: + +```shell-session +scp cache-private-key.pem root@cache:/var/secrets/cache-private-key.pem +``` + +Up until now, the binary cache daemon was in a restart loop due to the missing secret key file. +Check that it now works correctly: + +```shell-session +ssh root@cache systemctl status nix-serve.service +``` + +:::{important} +[](custom-binary-cache) using `cache-public-key.pem` on your local machine. +::: + +## Test availability + +The following steps check if everything is set up correctly and may help with identifying problems. + +### Check general availability + +Test if the binary cache, reverse proxy, and firewall rules work as intended by querying the cache: + +```shell-session +$ curl http://cache/nix-cache-info +StoreDir: /nix/store +WantMassQuery: 1 +Priority: 30 +``` + +### Check store object signing + +To test if store objects are signed correctly, inspect the metadata of a sample derivation. +On the binary cache host, build the `hello` package and get the `.narinfo` file from the cache: + +```shell-session +$ hash=$(nix-build '' -A pkgs.hello | awk -F '/' '{print $4}' | awk -F '-' '{print $1}') +$ curl "http://cache/$hash.narinfo" | grep "Sig: " +... +Sig: cache.example.org:GyBFzocLAeLEFd0hr2noK84VzPUw0ArCNYEnrm1YXakdsC5FkO2Bkj2JH8Xjou+wxeXMjFKa0YP2AML7nBWsAg== +``` + +Make sure that the output contains this line prefixed with `Sig:` and shows the public key you generated. + +(https-binary-cache)= +### Serving the binary cache via HTTPS + +If the binary cache is publicly accessible, it is possible to enforce HTTPS with [Let's Encrypt](https://letsencrypt.org/) SSL certificates. +Edit your `binary-cache.nix` like this and make sure to replace the example URL and mail address with yours: + +```{code-block} diff + services.nginx = { + enable = true; + recommendedProxySettings = true; +- virtualHosts.cache = { ++ virtualHosts."cache.example.com" = { ++ enableACME = true; ++ forceSSL = true; + locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}"; + }; + }; + ++ security.acme = { ++ acceptTerms = true; ++ certs = { ++ "cache.example.com".email = "you@example.com"; ++ }; ++ }; + + networking.firewall.allowedTCPPorts = [ + config.services.nginx.defaultHTTPListenPort ++ config.services.nginx.defaultSSLListenPort + ]; +``` + +Rebuild the system to deploy these changes: + +```shell-session +nixos-rebuild switch --no-flake --target-host root@cache.example.com +``` + +## Next steps + +If your binary cache is already a [remote build machine](https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds), it will serve all store objects in its Nix store. + +- [](custom-binary-cache) using the binary cache's hostname and the generated public key +- [](post-build-hooks) to upload store objects to the binary cache + +To save storage space, please refer to the following NixOS configuration attributes: + +- [`nix.gc`](https://search.nixos.org/options?query=nix.gc): Options for automatic garbage collection +- [`nix.optimise`](https://search.nixos.org/options?query=nix.optimise): Options for periodic Nix store optimisation + +## Alternatives + +- The [SSH Store](https://nix.dev/manual/nix/latest/store/types/ssh-store), [Experimental SSH Store](https://nix.dev/manual/nix/latest/store/types/experimental-ssh-store), and the [S3 Binary Cache Store](https://nix.dev/manual/nix/latest/store/types/s3-binary-cache-store) can also be used to serve a cache. + There are many commercial providers for S3-compatible storage, for example: + - Amazon S3 + - Tigris + - Cloudflare R2 + +- [attic](https://github.com/zhaofengli/attic): Nix binary cache server backed by an S3-compatible storage provider + +- [Cachix](https://www.cachix.org): Nix binary cache as a service + +## References + +- [Nix Manual on HTTP Binary Cache Store](https://nix.dev/manual/nix/latest/store/types/http-binary-cache-store) +- [`services.nix-serve` module options][`services.nix-serve`] +- [`services.nginx` module options][`services.nginx`] diff --git a/source/tutorials/nixos/index.md b/source/tutorials/nixos/index.md index be0d3dc60..0e044acd2 100644 --- a/source/tutorials/nixos/index.md +++ b/source/tutorials/nixos/index.md @@ -8,6 +8,10 @@ Learn how to configure, test, and install or deploy NixOS. - [](building-bootable-iso-image) - [](building-and-running-docker-images) +## Nix related services + +- [](binary-cache-setup) + ## Integration testing - [](integration-testing-using-virtual-machines) @@ -29,4 +33,5 @@ integration-testing-using-virtual-machines.md provisioning-remote-machines.md installing-nixos-on-a-raspberry-pi.md deploying-nixos-using-terraform.md +binary-cache-setup.md ``` diff --git a/source/tutorials/nixos/provisioning-remote-machines.md b/source/tutorials/nixos/provisioning-remote-machines.md index 241aa7645..ecd51f521 100644 --- a/source/tutorials/nixos/provisioning-remote-machines.md +++ b/source/tutorials/nixos/provisioning-remote-machines.md @@ -285,6 +285,12 @@ nixos-rebuild switch --no-flake --target-host root@target-host `nixos-anywhere` is not needed any more, unless you want to change the disk layout. + +# Next steps + +- [](binary-cache-setup) +- [](post-build-hooks) + ## References - [`nixos-anywhere` project page][`nixos-anywhere`]