WIP: src layout for python

[konstin]

For a mixed project with the current layout, python will always pick the local folder over the installed version, causing a number of problems (#335 (comment)). In pure python projects, this can be solved with the src layout. For rust projects, cargo already occupies src, so I've named the folder python_src (name open to bikeshedding). With this change pyo3-src-layout/pyo3_src_layout/__init__.py and pyo3-src-layout/python_src/pyo3_src_layout/__init__.py both work.

I hope this doesn't collide with #489.

Closes #335

[messense]

IMO python_src is a kinda ugly, why not just python?

[konstin]

My idea was this should work with something like the following layout:

my_tool
- src
  - lib.rs    Here's the core implementation
  - main.rs    The functionality exposed as cli
- python
  - src
    - lib.rs    The functionality exposed as pyo3 bindings
  - python_src
    - my_tool
      - __init__.py
      - addition_python_functionality.py
  - Cargo.toml     This crate contains the pyo3 bindings
- Cargo.toml    This crate contains the core functionality

This is similar to e.g. https://github.com/huggingface/tokenizers

I also thought that the _src suffix makes it clearer that the directory would be named src if it wasn't for cargo.

[konstin]

I had an idea for very different design which I think would be better.

The core idea is that for a pure rust project, it's a rust project (represented by Cargo.toml) but for a mixed project, it's a python project (represented by pyproject.toml) with an added rust submodule. For a pure rust project, still point -m to a Cargo.toml, for a mixed project point -m to a pyproject.toml. Mixed project would need to migrate to the following layout:

my-project
├── Readme.md
├── pyproject.toml
├── src
│   └── my_project
│       ├── __init__.py
│       └── bar.py
└── rust
    ├── Cargo.toml
    └── src
        └── lib.rs

This is more verbose than before, but it has the advantage of using the normal src layout for python while also having no confusion about the subfolder name (it's always rust). It also makes it clearer what is a submodule of what. The big disadvantage is that it would force everyone to restructure their projects.

[messense]

The core idea is that for a pure rust project, it's a rust project (represented by Cargo.toml) but for a mixed project, it's a python project (represented by pyproject.toml) with an added rust submodule. For a pure rust project, still point -m to a Cargo.toml, for a mixed project point -m to a pyproject.toml. Mixed project would need to migrate to the following layout:

my-project
├── Readme.md
├── pyproject.toml
├── src
│   └── my_project
│       ├── __init__.py
│       └── bar.py
└── rust
    ├── Cargo.toml
    └── src
        └── lib.rs

For existing Python projects adding a Rust extension to boost performance, this layout is quite nice. But for Rust projects adding a Python binding, it's a little annoying to have to move Rust source into a rust folder.

[davidhewitt]

I think the overall proposal is quite agreeable. Do you think it'd be possible to make this the default "mixed layout" and add configuration options to allow users to customise the project layout a bit if they wanted? e.g. I think in Cargo.toml you can specify the lib.path to customise the Rust location, maturin can presumably be compatible with that.

The big disadvantage is that it would force everyone to restructure their projects.

For existing Python projects adding a Rust extension to boost performance, this layout is quite nice. But for Rust projects adding a Python binding, it's a little annoying to have to move Rust source into a rust folder.

I agree with both of these points. I wonder if some automated commands e.g. maturin migrate-project-layout and maturin make-mixed-project could be written to ease the pain?

• migrate-project-layout it's possible to imagine a general command where the user can say where they want each of the Rust & python files to end up and maturin can do the shuffling around
• make-mixed-project could move the existing repo contents into a Rust subfolder and add a pyproject.toml, __init__.py etc

[clbarnes]

As has been said maturin's value to the whole spectrum from "rust project with thin python bindings" to "python project with a few lines of rust" will always introduce a conflict.

Every change breaks a workflow, of course, but in my projects I tend to rely on rust tooling for tracking versions etc. (cargo-release for releases, then the rust core stores CARGO_ENV_VERSION and python fetches it to find the version), because it's so much better than python's tooling. Would tucking Cargo.toml into a subdirectory make that harder?

[messense]

Closing in favor of #1185