Python: deprecate format and default to pyproject

[figsoda]

Motivation

Since the introduction of PEP 518 and some following PEPs that expands on the idea, the pyproject.toml format has gained a lot of adoption and became the de facto format for python packages.

However, a more popular option, and the current default format in nixpkgs is setuptools. Luckily, we might still be able to build these packages with format = "pyproject" because pyproject.toml falls back to using setuptools as its build backend as documented by pypa even if the package doesn't have a pyproject.toml. This might actually be preferred, because I ran into issues with format = "setuptools" when trying to package a setup.py package.

Proposal

I am proposing to deprecate the format option and replacing it with a pyproject option for packages that don't work with pyproject. We can do this over a longer period of time:

1. introduce a pyproject option that conflicts with the existing format option, where true is equivalent to format = "pyproject" and false is equivalent to format = "other"
   • python310Packages.buildPythonPackage: introduce pyproject option #254136
2. enforce newly added python packages to use the pyproject option instead of format
   • python310Packages.buildPythonPackage: introduce pyproject option #254136
3. convert existing format = "setuptools" and format = "pyproject" to pyproject = true and the rest to pyproject = false
4. deprecate format with a warning and make the default behavior pyproject = true (instead of format = "setuptools")
5. remove all pyproject = trues, as this is now the default
6. remove the format option altogether

Related Issues

• Python: remove format = "setuptools"; as default #124051
• pyproject build requires build-system definition nix-community/nix-init#119

[figsoda]

cc the python team: @FRidh @mweinelt @jonringer @tjni

[FRidh]

Yes, we should get rid of format, and this way forward sounds good. Although, I'd prefer if we right away consider also #170577 as that helps flattening our dependency tree, especially now since CA-derivations are well under way.

However, a more popular option, and the current default format in nixpkgs is setuptools. Luckily, we might still be able to build these packages with format = "pyproject" because pyproject.toml falls back to using setuptools as its build backend as documented by pypa even if the package doesn't have a pyproject.toml. This might actually be preferred, because I ran into issues with format = "setuptools" when trying to package a setup.py package.

We no longer use pip during build/install.

[pbsds]

A good chance to get rid of format = "flit".

I actually have a half decent pr to that end which I've yet to push. The only major blocker is the list of overrides defined by poetry2nix, which is massive.

[figsoda]

step 1: #254136

[FRidh]

A good chance to get rid of format = "flit".

I actually have a half decent pr to that end which I've yet to push. The only major blocker is the list of overrides defined by poetry2nix, which is massive.

I'd like us to remove format flit as soon as possible.

[pbsds]

I'd like us to remove format flit as soon as possible.

• python3Packages.flitBuildHook: remove #254309 😃

[mweinelt]

So format = "other" will be migrated to pyproject = false?

[figsoda]

So format = "other" will be migrated to pyproject = false?

Yes, format = "pyproject" | "setuptools" will become pyproject = true, and everything else will become pyproject = false

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-init-generate-nix-packages-from-urls-with-hash-prefetching-dependency-inference-license-detection-and-more/25035/18

[yajo]

Would it make sense to be able to point to the pyproject.toml file directly? So Nix could:

• Parse that TOML
• Read its build-backend.requirements
• If there's none, default to setuptools + wheel as PEP-518 explains.
• Try to find them by name from the current python3.pkgs

Hopefully most derivation definitions could be dramatically simplified.

[FRidh]

That would mean vendoring the pyproject.toml of projects in Nixpkgs since we do not want to have IFD.

[natsukium]

I'm not sure what to do with format = wheel.

Should we set pyproject = false and add wheelUnpackHook and pypaInstallHook to nativeBuildInputs for each package?

[mweinelt]

I think going for a boolean pyproject was a mistake and format enum better reflects the variety of options. The only thing that we would have needed to migrate is its default value, but that would've been breaking, so we're now taking this detour.

[natsukium]

I think going for a boolean pyproject was a mistake and format enum better reflects the variety of options.

Maybe so. However, I think it is a good idea to provide information about the hooks so that users can use them, not just for wheels.
In most cases (i.e. pyproject = true), they should be used by buildPythonPackage/Application, but sometimes I want to use them directly when using cmake or other build systems.
That is, if pyproject = false, then hooks can be used as needed.

[FlorianFranzen]

I am trying to package some python libs right now, so what is now the consensus?

IMHO the format field is clearly superior, since there are plenty of other formats out there worth supporting (egg, wheel, swig, skbuild, etc.)

[SuperSandro2000]

I am trying to package some python libs right now, so what is now the consensus?

pyproject = true;

since there are plenty of other formats out there worth supporting (egg, wheel, swig, skbuild, etc.)

I disagree. Supporting every format is just a maintenance burden especially if there is no real world usecase for it. Close to every package can use pyproject = true and the 2% other packages can use format other.

Also egg is very old, deprecated and replaced by wheel.
wheel is mostly used for binaries but we try to package from source builds. Any package that uses wheel for source code should be switched to the sdist or better directly downloaded from source.
IIRC swig has not so great cross support and all in tree usages I could spot on the fast where embedded in some other build system. It is probably not worth it to add it at that point.