Nix: No such file or directory: '~/.cache/pre-commit/...python3.10/bin/ruff'

[MatrixManAtYrService]

I'm working on a project which uses ruff via pre-commit. I've simplified the .pre-commit-config.yaml to the following:

---
repos:
  - repo: meta
    hooks:
      - id: check-hooks-apply
      - id: check-useless-excludes

  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: 'v0.0.229'
    hooks:
      - id: ruff

When I run pre-commit run --all-files I get the following:

 ❯ pre-commit run --all-files
[INFO] Initializing environment for https://github.com/charliermarsh/ruff-pre-commit.
[INFO] Installing environment for https://github.com/charliermarsh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Check hooks apply to the repository......................................Passed
Check for useless excludes...............................................Passed
ruff.....................................................................Failed
- hook id: ruff
- exit code: 1

[Errno 2] No such file or directory: '/home/matt/.cache/pre-commit/repo2mikycmg/py_env-python3.10/bin/ruff'

Which is weird, because that file definitely exists.

This is on my Linux machine (NixOS 23.05). I've been working around it by just doing the linting on macOS, but that's kind of a pain. Any idea what I can do to fix this?

[charliermarsh]

Could this be related to astral-sh/ruff#1699?

[MatrixManAtYrService]

Seems likely, I have that same experience:

❯ nix-shell -p python311
[nix-shell:~/2023/01/21]$ python -m venv venv
[nix-shell:~/2023/01/21]$ source venv/bin/activate

[nix-shell:~/2023/01/21]$ pip install ruff
Collecting ruff
  Using cached ruff-0.0.229-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Installing collected packages: ruff
Successfully installed ruff-0.0.229

[nix-shell:~/2023/01/21]$ ruff
bash: /home/matt/2023/01/21/venv/bin/ruff: No such file or directory

I can add the nixpkgs ruff to my flake.nix, which allows me to call ruff directly in a devshell, but I'm not sure how to convince pre-commit to use that one instead of the one in its cache.

[AntonLydike]

There is a workaround, but it's really really ugly:

rm '<offending ruff binary>'
ln -s $(which ruff) '<offending ruff binary>'

[TheLostLambda]

That workaround works, but obviously only until pre-commit regenerates that cache! It would be nice a more proper fix for this at some point :)

[anund]

This is indeed related to astral-sh/ruff#1699 and doesn't have a fix here. (I missed the repo context while writing my original suggestion)

The contract for a pre-commit python language hook is invoking pip install. The binary pulled from pip has a dynamic dependency on libgcc and a elf interpreter of /lib64/ld-linux-x86-64.so.2. On nix neither of those things are guaranteed to be where they are conventionally assumed to be on linux distros using ~loosely FHS based file system organizations. The default error message in the case of a binary that doesn't load correctly on linux is 'No such file or directory' rather than "something can't be loaded in the elf metadata".

Nix specific suggestion for people coming from google

The beginnings of a fix might look like invoking autoPatchelf in the shell hook.

An example .pre-commit-config.yaml that uses ruff

---
repos:
  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: 'v0.0.229'
    hooks:
      - id: ruff

An example flake.nix that add pre-commit and a libgcc which ruff binary depends on. Note I don't regularly use flakes, this works but may be bad copy+pasta. The important bits are in the shellhook and in nativeBuildInputs

{
  description = "Dev env";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    flake-compat.url = "github:edolstra/flake-compat";
    flake-compat.flake = false;
  };
  outputs = {
    nixpkgs,
    flake-utils,
    ...
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
        overlays = [];
      };
    in {
      devShells.default = pkgs.mkShell {
        name = "dev-env";
        buildInputs = with pkgs; [
          pre-commit
          stdenv.cc.cc.libgcc
        ];
        nativeBuildInputs = [ pkgs.autoPatchelfHook ];
        shellHook = ''
          pre-commit install

          # oddly the python pre-commit hook doesn't actually install things during install
          # here's a hack to force ruff into the pre-commit cache (works but is not great)
          pre-commit run

          # pre-commits output is a bit verbose when it succeeds, choose your own output wrangling
          patch=$(autoPatchelf ~/.cache/pre-commit/)
          if [[ $? -ne 0 ]]; then
            echo "$patch"
            exit 1
          else
            echo "patched ~/.cache/pre-commit"
          fi
        '';
      };
    });
}

There are probably a number of ways to improve how this is handled. Improvements could include: a better way to get the pre-commit hook to actually install during 'install', better output wrapping, a better understanding of how/when pre-commit chooses to install things to ~/.cache/pre-commit, a better place to add ~shared devshell preambles like this.
Inspiration from https://sandervanderburg.blogspot.com/2018/10/auto-patching-prebuilt-binary-software.html

</details>
(mistakenly assumed this repo was nix)

[anund]

I believe this issue can be closed and, perhaps, tracked somewhere more nix specific.

[MatrixManAtYrService]

I agree. This repo is a way to access the problem, but it doesn't itself have a problem. Let's close it.

The approach in @anund's post works on NixOS, but autopatchelf breaks on darwin, so you have to do some kind of conditional logic to ensure that it doesn't attempt to patch the binary in that case--which is fine, because the ruff build doesn't break on darwin.

[MatrixManAtYrService]

If I'm understanding correctly, the "right way" to fix this requires:

1. somebody who cares about nix...
2. ...to go fix the way pip (a python package manager) installs ruff (a program written in rust) such that it depends on a packaged libgcc instead of just loading whatever it finds lying around in a conventional location (and breaking when it doesn't find it).

But a big part of the reason to be a 1-type person is because you don't want to deal with 2-type problems anymore. Maybe our hero is out there. Maybe they'll fix it the right way. But I'm probably not that hero, so I'm going to seek a better workaround--one that doesn't involve asking pip to package rust binaries in the first place.

My initial explorations of https://github.com/cachix/pre-commit-hooks.nix indicate that it wants to be configured in a way that would break pre-commit for other contributors to the repo. It will say things like this:

pre-commit-hooks.nix: WARNING: Refusing to install because of pre-existing .pre-commit-config.yaml

1. Translate .pre-commit-config.yaml contents to the new syntax in your Nix file
   see https://github.com/cachix/pre-commit-hooks.nix#getting-started
2. remove .pre-commit-config.yaml
3. add .pre-commit-config.yaml to .gitignore

My team would not like it if I made that change (they're not all nix users). But perhaps...

A. a ruff hook can be added to pre-commit-hooks.nix
B. pre-commit-hooks.nix can be... configured? forked? such that it can comfortably share a repo with pre-commit, and instead just overrides certain hooks with ones that use a ruff in the nix store.

This would solve my problem, but maybe it's overkill? Maybe it won't solve your problem? I'm curious if others think it's a sane approach.

[NickLarsenNZ]

For anyone coming here still looking for a solution, here's what worked for me:

1. Delete the cached ruff (eg: if you ran the hook before installing it via nixpkgs)
   eg: rm /home/nick/.cache/pre-commit/repoz9ecmeyb/py_env-python3.11/bin/ruff
2. Rerun pre-commit (it will not used the cached one, and instead use your installed and correctly linked binary).