pre-commit clobbers python311 PYTHONPATH entries with references to python310 packages

[MatrixManAtYrService]

Describe the bug

If you install pre-commit alongside python311, the python311 site-packages directory disappears from the PYTHONPATH, which (I think) causes other packages to break.

nix-shell --pure -p pkgs.python311 pkgs.pre-commit has PYTHONPATH entries:

/nix/store/m006rlxdxfdjvcnp7a9dq63k9g7rlbab-python3.10-pre-commit-2.20.0/lib/python3.10/site-packages
/nix/store/7nmn7h4izvx59jj3bx03rii4gxvf5jcw-python3.10-cfgv-3.3.1/lib/python3.10/site-packages
/nix/store/nrbblrkb8qzk04wpjabxyrzcgj7kmms2-python3.10-six-1.16.0/lib/python3.10/site-packages
/nix/store/al6g1zbk8li6p8mcyp0h60d08jaahf8c-python3-3.10.9/lib/python3.10/site-packages
/nix/store/76lc8r362zijsrax0hqy6q255viqpdfb-python3.10-identify-2.5.12/lib/python3.10/site-packages
/nix/store/w3dyvcw8lmc2kxf5m9f48qaqjfbljwjw-python3.10-nodeenv-1.7.0/lib/python3.10/site-packages
/nix/store/6zh035j7znhpb33qcxyg8jbxxham5hry-python3.10-setuptools-65.3.0/lib/python3.10/site-packages
/nix/store/ms7lw8c9zm8fqx1h9614d4jri54dl8ld-python3.10-pyyaml-6.0/lib/python3.10/site-packages
/nix/store/aqgjg15az5da35ifv9rxr03355qjqdmq-python3.10-toml-0.10.2/lib/python3.10/site-packages
/nix/store/y4rvs95nml9prcl7bvv46k6s526i4zc5-python3.10-virtualenv-20.16.5/lib/python3.10/site-packages
/nix/store/h8nlfjirwcfz5y5khirgapjwhnzx4j8k-python3.10-backports-entry-points-selectable-1.2.0/lib/python3.10/site-packages
/nix/store/8hd25px048a3fsiw0nw17vy9yc987mxl-python3.10-distlib-0.3.6/lib/python3.10/site-packages
/nix/store/z91md4v0zn19l14ikjqng1a67x2kcynp-python3.10-filelock-3.8.0/lib/python3.10/site-packages
/nix/store/k0hgyiipf0rd8ccgg1kzx419jb4h69s4-python3.10-platformdirs-2.5.3/lib/python3.10/site-packages

Notice that the above list does not include any python 3.11 entries. To get those you have to remove pre-commit (nix-shell --pure -p pkgs.python311):

/nix/store/xscl8kqmywwjvqv7l9i5j8xb4a9b655y-python3-3.11.1/lib/python3.11/site-packages

The actual problem that lead me here had to do with pkgs.python311Packages.psycopg being unable to satisfy its runtime requirements. The problem went away when I removed pkgs.pre-commit.

I've documented that separately, but haven't filed it as an issue yet. I'm not 100% sure that PYTHONPATH is the issue, but it does seem like pre-commit is being a noisy neighbor in some way or another.

Steps To Reproduce

Contrast the output of these commands:

nix-shell --pure -p pkgs.python311 pkgs.pre-commit
echo $PYTHONPATH

with the output of these...

nix-shell --pure -p pkgs.python311
echo $PYTHONPATH

Expected behavior

I'm not sure, but whatever it is I don't think that installing pre-commit should hide python3.11/site-packages

Notify maintainers

@borisbabic

Metadata

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.86, NixOS, 23.05 (Stoat), 23.05pre442253.befc83905c9`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.12.0`
 - channels(matt): `"nixos"`
 - channels(root): `"home-manager, nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[MatrixManAtYrService]

I'm working around this by just commenting pkgs.pre-commit in my flake.nix whenever I want to run my app, and then re-adding it whenever I want to run pre-commit.

        devShells.default = pkgs.mkShell {
          packages = [
            poetry2nix.packages.${system}.poetry
            pkgs.python311Packages.psycopg
            pkgs.python311
            #pkgs.pre-commit
          ];

If there's a smarter workaround I'd love to know about it.

[austinbutler]

In my case it's sort of the opposite. I had a Python env (python310.withPackages) using Python 3.10 with pre-commit together in a mkShell and that worked fine. I bumped the env to Python 3.11 recently and that made pre-commit stop working: /nix/store/412d25bdawy2idxfpnis8vjcm8cz2ag7-python3-3.10.11/bin/python3.10: Error while finding module specification for 'pre_commit.meta_hooks.check_hooks_apply' (ModuleNotFoundError: No module named 'pre_commit').

When I printed the PYTHONPATH just before this line was run I noticed everything references Python 3.11, even though pre-commit was running via Python 3.10. That gave me the hint to roll back the env to 3.10, which fixes pre-commit.

[MatrixManAtYrService]

@fabaff I'm sorry to apply "touched it last" logic here, but I can see that you're active and you probably know a bit more about how this works than I do. Would you be willing to share any thoughts you have about this?

[Artturin]

this needs to be done

nixpkgs/pkgs/tools/package-management/poetry/default.nix

Lines 49 to 53 in fc7d9a1

	# Propagating dependencies leaks them through $PYTHONPATH which causes issues
	# when used in nix-shell.
	postFixup = ''
	rm $out/nix-support/propagated-build-inputs
	'';

[MatrixManAtYrService]

That's very helpful, thanks @Artturin. I'll attempt a PR to fix this, hopefully this weekend.

[skaphi]

I get a similar error using nix develop on this flake

{
  inputs.nixpkgs.url = "nixpkgs/nixos-23.05";

  outputs = { self, nixpkgs }:
    let
      x86_pkgs = nixpkgs.legacyPackages."x86_64-linux";
    in
    {
      devShells."x86_64-linux" = {
        default = x86_pkgs.mkShell {
          packages = with x86_pkgs; [
            black
            python311
          ];
        };
      };
    };
}

$ python3
Python 3.10.12 (main, Jun  6 2023, 22:43:10) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Weirdly, I cannot reproduce it with nix-shell -p python311 black