Cache /nix/store between jobs

[arximboldi]

Hi!

Sorry if this is a stupid question, I am new to using GitHub Actions.

I would like to cache /nix/store between builds to speed up the setup. Is it as easy as following this:
https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#caching-dependencies
or are there any caveats or pitfalls I should take into account?

[domenkozar]

See https://nix.dev/tutorials/continuous-integration-github-actions.html

[arximboldi]

Hmm, I'm giving that a try, but as far as I understand, that will use Cachix as a binary cache for stuff that is build from source. In my case, I just want to save time downloading binaries from Nixpkgs, keeping already a local copy using the Github Actions cache store. Will Cachix also do that? (Does not look like it from looking at it's code, but I'm not sure.)

[domenkozar]

That's not much of an improvement, either way all that content needs to be downloaded to the runner. Unless you self-host the runner.

[arximboldi]

Ok, thank you for your reponse! I am using Cachix now and it has still cut a couple of minutes off my actions. Thanks a lot for the great work supporting this great service and your overall contributions to the Nix ecosystem! ❤️

[domenkozar]

❤️

[robinp]

For posterity: FYI this is the setup I came up with that (currently) works to cache the plain /nix/store. Note, the cache key is too simplistic, you should include the hash of your .nix files maybe (with fallback key prefix specified).

But it overcomes: 1) the permission errors you get when trying to untar the stored cache on the second run, and 2) the storing part trying to persist lock files and else which will result in permission errors.

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/[email protected]

    - run: |
        # Create with liberal rights, otherwise cache action will complain
        # about permission errors.
        sudo mkdir -p /nix/store
        sudo chmod -R 777 /nix

    - name: Cache nix env take N+1
      uses: actions/cache@v2
      with:
        path: |
          # See https://github.com/actions/cache/pull/726
          /nix/store/**
          # Missing something?
          /nix/var/nix/*/*
          /nix/var/nix/db/*
          /nix/var/nix/db/*/**
          !/nix/var/nix/daemon-socket/socket
          !/nix/var/nix/userpool/*
          !/nix/var/nix/gc.lock
          !/nix/var/nix/db/big-lock
          !/nix/var/nix/db/reserved
        key: ${{ runner.os }}-nix-store

    - uses: cachix/install-nix-action@v15
      with:
        nix_path: nixpkgs=channel:nixos-unstable

    - name: Install custom nix env
      run: nix-env -f my-build.nix -i '.*'

[arximboldi]

That's pretty cool, thanks @robinp !

[hughjfchen]

For posterity: FYI this is the setup I came up with that (currently) works to cache the plain /nix/store. Note, the cache key is too simplistic, you should include the hash of your .nix files maybe (with fallback key prefix specified).

But it overcomes: 1) the permission errors you get when trying to untar the stored cache on the second run, and 2) the storing part trying to persist lock files and else which will result in permission errors.

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/[email protected]

    - run: |
        # Create with liberal rights, otherwise cache action will complain
        # about permission errors.
        sudo mkdir -p /nix/store
        sudo chmod -R 777 /nix

    - name: Cache nix env take N+1
      uses: actions/cache@v2
      with:
        path: |
          # See https://github.com/actions/cache/pull/726
          /nix/store/**
          # Missing something?
          /nix/var/nix/*/*
          /nix/var/nix/db/*
          /nix/var/nix/db/*/**
          !/nix/var/nix/daemon-socket/socket
          !/nix/var/nix/userpool/*
          !/nix/var/nix/gc.lock
          !/nix/var/nix/db/big-lock
          !/nix/var/nix/db/reserved
        key: ${{ runner.os }}-nix-store

    - uses: cachix/install-nix-action@v15
      with:
        nix_path: nixpkgs=channel:nixos-unstable

    - name: Install custom nix env
      run: nix-env -f my-build.nix -i '.*'

I'd followed this but always got the following error when saving the nix packages to the cache
after having my project built successfully:
Warning: Maximum call stack size exceeded

[silviogutierrez]

@robinp I couldn't get your snippet to work, but had some luck with this (Edited, mentioned the wrong person):

- name: Cache nix env take N+1
  uses: actions/cache@v2
  id: nix-cache
  with:
      path: |
          /tmp/nix-cache
      key: ${{ runner.os }}-nix-cache
- name: cache hit
  if: steps.nix-cache.outputs.cache-hit == 'true'
  run: |
      sudo chown -R root /tmp/nix-cache

      sudo cp -fRT /tmp/nix-cache/ /nix/
      # cp sometimes errors out on MacOS, but rsync works.
      # sudo rsync -a /tmp/nix-cache/ /nix/
      sudo pkill nix-daemon
- name: do stuff
  run: nix-shell --command 'echo OK'
- run: |
      sudo cp -r /nix /tmp/nix-cache
      # cp sometimes errors out on MacOS, but rsync works as below.
      # sudo rsync -a --exclude .Trashes --exclude /var/nix/daemon-socket /nix/ /tmp/nix-cache/
    
      sudo rm -rf /tmp/nix-cache/var/nix/daemon-socket/socket
      sudo rm -rf /tmp/nix-cache/var/nix/userpool
      sudo rm -rf /tmp/nix-cache/var/nix/gc.lock
      sudo rm -rf /tmp/nix-cache/var/nix/db/big-lock
      sudo rm -rf /tmp/nix-cache/var/nix/db/reserved

Basically, I gave up on trying to cache /nix directly as it's too special of a directory. Instead, I make an intermediary copy, then cache that. On cache hit/restore, I manually copy over stuff.

It worked great on Linux, but was really slow on MacOS. Especially creating the intermediary copy (which is absolutely necessary on MacOS, as /nix is a special volume on MacOS).

Tragically, it's on MacOS where I need the speed up the most, and this didn't really help too much. On Linux, for my projects, it's almost not worth the complexity.

[silviogutierrez]

I wish nix had a first-class way to get just the stuff we need to cache. I can't quite tell if this is helpful: https://fzakaria.com/2020/08/11/caching-your-nix-shell.html since I'm not actually compiling any binaries, really just downloading stuff.

Or this: https://docs.cachix.org/faq#is-there-a-way-to-cache-nix-shell

Both seem to help only when compiling stuff. I previously discussed this with @domenkozar on cachix/cachix-action#61 , but it was a while back. I wonder if things have changed since then.

[ruuda]

If you want to cache the Nix store with the GitHub Actions cache, instead of trying to work around the challenges of caching the raw /nix/store, there is a simpler way that also produces smaller caches: nix-store --{import,export}.

- name: "Cache Nix store"
  uses: actions/[email protected]
  id: nix-cache
  with:
    path: /tmp/nixcache
    key: "FIXME: Pick a cache key suitable for your use case"

- name: "Install Nix"
  uses: cachix/install-nix-action@v17
  with:
    install_url: "https://releases.nixos.org/nix/nix-2.11.0/install"

- name: "Import Nix store cache"
  if: "steps.nix-cache.outputs.cache-hit == 'true'"
  run: "nix-store --import < /tmp/nixcache"

- name: "Enable Cachix"
  uses: cachix/cachix-action@v10
  with: "FIXME: Fill in your details"

- name: "Build application"
  run: "nix build --print-build-logs"

- name: "Export Nix store cache"
  if: "steps.nix-cache.outputs.cache-hit != 'true'"
  run: "nix-store --export $(find /nix/store -maxdepth 1 -name '*-*') > /tmp/nixcache"

It works reasonably well in combination with Cachix; I’ve found that for some derivations that need to get hundreds of small-ish store paths from Cachix, it can take a few minutes to fetch them individually, while downloading from the GitHub Actions cache + importing into the Nix store together took less than a minute. The downside is that you need to watch out for what is in your /nix/store when the cache entry gets produced. For example, if your build is a no-op when the GitHub Actions cache is filled, because the result was in Cachix, then the build dependencies will not be realized, and they will be absent from the GitHub Actions cache.

[aabccd021]

If you use flake, you could use experimental nix copy command instead of nix-store --{import,export}.

For example, if you want to cache your devShell

flake.nix:

{
  outputs = { nixpkgs }: with nixpkgs.legacyPackages.x86_64-linux; {
    devShell.x86_64-linux = mkShell {
      buildInputs = [
        nodejs-16_x
      ];
    };
  };
}

You could do:

- name: "Cache Nix store"
  uses: actions/[email protected]
  id: nix-cache
  with:
    path: /tmp/nixcache
    key: "FIXME: Pick a cache key suitable for your use case"

- name: "Install Nix"
  uses: cachix/install-nix-action@v17
  with:
    install_url: "https://releases.nixos.org/nix/nix-2.11.0/install"

- name: "Import Nix store cache"
  if: "steps.nix-cache.outputs.cache-hit == 'true'"
  run: "nix copy --from /tmp/nixcache ./#devShell.x86_64-linux"

- name: "Enable Cachix"
  uses: cachix/cachix-action@v10
  with: "FIXME: Fill in your details"

- name: "Build application"
  run: "nix build --print-build-logs"

- name: "Export Nix store cache"
  if: "steps.nix-cache.outputs.cache-hit != 'true'"
  run: "nix copy --to /tmp/nixcache ./#devShell.x86_64-linux"

[deemp]

Caching /nix/store directly worked for me in https://github.com/nix-community/cache-nix-action

[drupol]

Anybody has something equivalent for Gitlab CI ?