build: add poetry tooling

[fleimgruber]

No description provided.

[codecov]

Codecov Report

Merging #849 (870e7d8) into dev (c494ee7) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #849   +/-   ##
=======================================
  Coverage   97.29%   97.29%           
=======================================
  Files          43       43           
  Lines        4028     4028           
=======================================
  Hits         3919     3919           
  Misses        109      109           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c494ee7...870e7d8. Read the comment docs.

[cosmicBboy]

hey @fleimgruber great initiative on this PR!

One question I have: as you can see here, the conda environment file is the source of truth for all the dev dependencies of pandera, and this script ensure that the requirements.txt file is equivalent to the env.yml file.

Is there similar tooling for syncing setup.py and pyproject.toml files, OR is there a way we can just use the pyproject.toml file and do away with setup.py? I'm not too familiar with poetry and the associated toolchain, but my main concern/objective is to minimize burden of maintaining two separate files. E.g. is there a way of just specifying pyproject.toml and using that to pip install and poetry install pandera?

[cosmicBboy]

And to ask the converse question: is it not possible to install setup.py-based projects on poetry?

[cosmicBboy]

This discussion seems relevant: python-poetry/poetry#1135

To merge this PR, I'd like to fulfill these requirements:

• A single file that declares the project build, dependencies, and metadata (alternatively, a solution that auto-generates e.g. the pyproject.toml file from a single source of truth).
• Support for dynamically loading the project version (see here)
• Supports editable installs (like pip install -e .) for ease of development experience

[fleimgruber]

Thanks for your comments and questions @cosmicBboy.

E.g. is there a way of just specifying pyproject.toml and using that to pip install and poetry install pandera?

Installation with pip from pyproject.toml seems to be experimental for now: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

is there a way we can just use the pyproject.toml file and do away with setup.py?

Given the above, not at the moment AFAICS.

Is there similar tooling for syncing setup.py and pyproject.toml files?

I am not aware of any.

my main concern/objective is to minimize burden of maintaining two separate files

This is very reasonable and I was expecting it.

the conda environment file is the source of truth for all the dev dependencies of pandera

There is an inactive issue for creating the pyproject.toml form a conda env file: conda/conda#10633.
At some point there was dephell (https://github.com/dephell/dephell), but I guess it was too ambitious.
There are some ways to generate the setup.py from the pyproject.toml file:

• Support generation of poetry manged setup.py file python-poetry/poetry#761 (comment)
• https://github.com/sdss/flicamera/blob/main/create_setup.py
• https://github.com/abersheeran/poetry2setup

Support for dynamically loading the project version

I guess this won't be done as pyproject.toml is seen as the source of truth for this.
Some more discussion on the topic (note the OP here): python-poetry/poetry#4299.

Supports editable installs (like pip install -e .) for ease of development experience

This is possible and done automatically with recent versions of poetry and one reason it makes it so convenient for development.

A single file that declares the project build, dependencies, and metadata (alternatively, a solution that auto-generates e.g. the pyproject.toml file from a single source of truth).

I guess maintaining two files is not worth it at the moment. We can revisit this when Setuptools finally moves to pyproject.toml. But even then it might be good to have setup.py to be backwards compatible.

If you do not want to accept this PR for the valid stated reasons (and I am with you on those), I will just keep my local pyproject.toml around and keep it in sync with project development - it works for me, I just wanted to share so that people can use it locally as well.

[jeffzi]

In addition to @fleimgruber's detailed answer, I'd like to point out that poetry does not support PEP 621, which standardizes pyproject.toml. Dependencies management, in particular, would be a blocker for us:

setuptools (standard pyproject.toml):

# pyproject.toml
[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[project]
name = "my_package"
description = "My package description"
dependencies = [ 'importlib-metadata; python_version<"3.8"']

[project.optional-dependencies]
rest = ["requests"]

poetry:

# pyproject.toml
[build-system]
requires = ["poetry>=0.1.4"]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "my_package"
description = "My package description"

[tool.poetry.dependencies]
importlib-metadata = { version = "^4.11.3", python = "<3.8" }

[tool.poetry.dev-dependencies]
rest = ["requests"]

It's not clear when poetry will be compliant with PEP 621: python-poetry/roadmap#3
When it is, developers should be able to modify the [build-system] section to switch to their preferred tool.

Note that flit is PEP 621 compliant but I have no experience with it.

re: conda. I found conda-lock that can create conda lock files from pyproject.toml. Conda already supports requirements.txt (poetry can export it), but I heard it's prone to errors.

Long term, it seems that standard PEP 621 pyproject.toml is the way to go but there are still many moving parts.

[cosmicBboy]

Cool, thanks for your initiative on this @fleimgruber ! Let's table this for now until there's better compatibility with setuptools. Gonna close this issue for now, but will keep #848 open.