Add option to build just the compiled extension modules

[ofek]

Hello! I'm starting an effort to make it easier and more standardized to build extension modules in Python. This was discussed recently on the Discourse Python forum but essentially the end goal is something like what is documented here and eventually a PEP. My view is that the Python community has not fully completed the expressed outcome of PEP 517 in that build backends are still (for the most part) reliant on setuptools for building non-Python code bases.

In my mind, there is a component that selects what files are to be included in an archive which we call the build backend and some other component that interacts with a compiler to generate some files. I think that there is no reason at all that they must be the same component.

In this hypothetical future any build backend would be able to trigger the building of extension modules based on user configuration. So in the case of maturin, that would look like (forgetting about what the final interface might be, that's my job) a way to just build the compiled bits and output the location of the files (and maybe some metadata) whether that be with a direct function call or via CLI.

I'm not too familiar with this code base yet but I think it wouldn't be too challenging to add such an option. Is that an accurate assessment?

Also, please let me know what you think of this general proposal. If it happens, you wouldn't actually need to maintain your own logic for building wheels and source distributions, etc. only the Rust part. Of course you can always still provide that functionality if you wanted.

Thank you for your time 🙂

[messense]

Sounds interesting, I'll keep an eye on it.

I'm not too familiar with this code base yet but I think it wouldn't be too challenging to add such an option. Is that an accurate assessment?

We could add a build-ext command similar to the setuptools build_ext command to only build the shared library, not too challenging to add but definitely needs a bit refactoring to decouple it from wheel building.

[konstin]

Building the extension itself is the easy part, on linux even just cargo build will produce an .so that you only need to copy. I would actually have preferred for maturin to be a poetry or hatch plugin! They offer all the features around environments, dependencies, etc. that users want and need that maturin could can't (and kinda also shouldn't) try to compete with. (Another potential solution would be to go the npm/cargo route and just make it comfortable to have multiple local packages and in-repo path-dependencies, so that e.g. maturin could stop shipping python code and just do a minimal package with the shared library and everything else is done in the main project with the "real" packaging tool. There's definitely design space beyond the currently limitations of python packaging to explore!)

The hard part is configuration: Where is the Cargo.toml and where should the .so go? What are the options to compile: Debug or release, are we cross compiling and which cargo rustc options to add? How do we know when we need to recompile? Which configuration files contain this information and how can the user overwrite them? This is something that PEP 517 skips over by saying config_settings is a kind of **kwargs that may have semantics added (this is already a problem today because e.g. PEP 517 build with maturin will always be release builds because we have no standard way to let the user set a flag for that). In my eyes, the solution needs to work with some kind of schema that's stricter than **kwargs (or worse, arbitrary python code), be integrated with pyproject.toml, have a story for error handling, and work with the cli of pip, other frontends such as python -m build and calling tools such as maturin directly.

For my understanding, PEP 517 is a sdist -> whl spec, and it does this job mostly really well (again except for configuration, which isn't a problem normally if you just want to pip install). What you are proposing to me is different goal, that is a standardized poetry/hatch/etc. plugin api, which would be really cool to have :)

On the practical side, exposing a config json -> built shared library either as cli or with pyo3 shouldn't be too hard. You might want to look at BuildOptions (the user input) or BuildContext (the resolved configuration) to get an impression for the API.

[flying-sheep]

Hi, I’d like to use pyo3-stub-gen together with maturin’s github action.

This doesn’t seem possible without wasting time:

• maturin isn’t available as a hatch plugin so I can’t combine it with other build steps as part of the python build process
• I can’t run arbitrary code inside of the maturin container
• I could of course manually run cargo run --bin=stub_gen, but that would compile my library an additional time before running it

So splitting maturin into a core library that can be wrapped into a hatchling plugin would be amazing!