Add support for PEP-631 (PEP 621)

[jancespivo]

What's the problem this feature will solve?

PEP-631 describes dependencies can be declared in pyproject.toml under [project.dependencies] section (see https://www.python.org/dev/peps/pep-0631/#dependencies):

[project]
dependencies = [
...
]

It is tool-independent way how to describe dependencies instead of tool-specific ones like [tool.poetry.dependencies]

Describe the solution you'd like

Add support for https://www.python.org/dev/peps/pep-0631.

Alternative Solutions

Additional context

[webknjaz]

I think that this is a duplicate of some old FRs but I cannot find those issues I seem to recall.

[AndydeCleyre]

EDIT: I guess I was wrong. Running pip-compile pyproject.toml appeared to be working but I think was using the setup.cfg that the test project had, or something...

---

Original, wrong, comment below:

---

AFAIU this is already supported in terms of using pyproject.toml as a source, so this request is specifically about using pyproject.toml as a destination, right?

But then, the requirements you'd put there are IMO more likely and more reasonably equivalent to the "loose" requirements defined in your *.in files, rather than the compiled result. I do have a zsh function pypc which populates a pyproject.toml with the *.in entries, recently demoed at #1495.

[AndydeCleyre]

So this is a request to support the new spec in the file as a source, then right?

[AndydeCleyre]

Some notes:

• we currently use this to intake setup files including pyproject.toml:

from pep517 import meta
dist = meta.load(os.path.dirname(os.path.abspath(src_file)))

• discussion at Avoid generating metadata in pip download --no-deps ... pypa/pip#1884 (comment):

No-one has yet looked at PEP 621 support in pip - apart from the obvious "no-one has been interested enough" reason, I think it's a matter of waiting until enough projects publish PEP 621 compatible metadata (and specifically dependency metadata) to make it worthwhile. Personally, I think that when setuptools starts writing PEP 621 data is probably the point at which I'd see it as worthwhile for pip (there will be a lot of projects that won't be able to generate static dependencies, even if setuptools supports them, but at least having the possibility seems like a reasonable starting point). Of course, you might feel differently about whether we should wait on a particular backend

• See also: Discussion: support for pyproject.toml configuration pypa/setuptools#1688

[jancespivo]

So this is a request to support the new spec in the file as a source, then right?

Yes, exactly.

[jancespivo]

Original PEP-631 has been accepted and was merged into PEP 621. It is the same however it might add some context:
https://peps.python.org/pep-0621/#dependencies-optional-dependencies

[AndydeCleyre]

Can we get a good example, with expected vs actual output?

[jancespivo]

Can we get a good example, with expected vs actual output?

Currently there is no output when dependencies are declared in pyproject.toml under [project.dependencies] section. The section is ignored by pip-tools completely.

Expected output would be the same as if dependencies was declared under [tool.poetry.dependencies] section (which is already implemented in pip-tools).

[AndydeCleyre]

@jancespivo

I'm asking for a good example, because my current test case is a bad example, in that it seems to work:

$ git clone https://github.com/AndydeCleyre/ptrender
$ cd ptrender
$ python3 -m venv venv
$ . ./venv/bin/activate
$ pip install -U pip-tools
$ cat pyproject.toml

[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "ptrender"
authors = [{name = "Andy Kluger"}]
readme = "README.rst"
license = {file = "LICENSE"}
keywords = ["pyratemp", "template", "render", "renderer"]
requires-python = ">=3.6"
dynamic = ["version", "description"]
dependencies = ["plumbum", "pyratemp"]
classifiers = [
    "Topic :: Text Processing :: Markup",
    "Topic :: Utilities",
]

[project.urls]
Home = "https://github.com/andydecleyre/ptrender"

[project.optional-dependencies]
yaml = ["strictyaml"]
dev = ["flit", "ipython"]

[project.scripts]
ptrender = "ptrender.ptrender:TemplateRenderer"
vwrite = "ptrender.ptrender:VWriter"

$ pip-compile --annotation-style=line --no-header -o - pyproject.toml

plumbum==1.7.2            # via ptrender (pyproject.toml)
pyratemp==0.3.2           # via ptrender (pyproject.toml)

$ pip-compile --annotation-style=line --no-header -o - --extra yaml pyproject.toml

plumbum==1.7.2            # via ptrender (pyproject.toml)
pyratemp==0.3.2           # via ptrender (pyproject.toml)
python-dateutil==2.8.2    # via strictyaml
six==1.16.0               # via python-dateutil
strictyaml==1.6.1         # via ptrender (pyproject.toml)

$ pip-compile --annotation-style=line --no-header -o - --extra dev pyproject.toml

asttokens==2.0.5          # via stack-data
backcall==0.2.0           # via ipython
certifi==2022.5.18.1      # via requests
charset-normalizer==2.0.12  # via requests
decorator==5.1.1          # via ipython
docutils==0.18.1          # via flit
executing==0.8.3          # via stack-data
flit==3.7.1               # via ptrender (pyproject.toml)
flit-core==3.7.1          # via flit
idna==3.3                 # via requests
ipython==8.3.0            # via ptrender (pyproject.toml)
jedi==0.18.1              # via ipython
matplotlib-inline==0.1.3  # via ipython
parso==0.8.3              # via jedi
pexpect==4.8.0            # via ipython
pickleshare==0.7.5        # via ipython
plumbum==1.7.2            # via ptrender (pyproject.toml)
prompt-toolkit==3.0.29    # via ipython
ptyprocess==0.7.0         # via pexpect
pure-eval==0.2.2          # via stack-data
pygments==2.12.0          # via ipython
pyratemp==0.3.2           # via ptrender (pyproject.toml)
requests==2.27.1          # via flit
six==1.16.0               # via asttokens
stack-data==0.2.0         # via ipython
tomli==2.0.1              # via flit
tomli-w==1.0.0            # via flit
traitlets==5.2.1.post0    # via ipython, matplotlib-inline
urllib3==1.26.9           # via requests
wcwidth==0.2.5            # via prompt-toolkit

# The following packages are considered to be unsafe in a requirements file:
# setuptools

$ pip-compile --annotation-style=line --no-header -o - --extra dev,yaml pyproject.toml

asttokens==2.0.5          # via stack-data
backcall==0.2.0           # via ipython
certifi==2022.5.18.1      # via requests
charset-normalizer==2.0.12  # via requests
decorator==5.1.1          # via ipython
docutils==0.18.1          # via flit
executing==0.8.3          # via stack-data
flit==3.7.1               # via ptrender (pyproject.toml)
flit-core==3.7.1          # via flit
idna==3.3                 # via requests
ipython==8.3.0            # via ptrender (pyproject.toml)
jedi==0.18.1              # via ipython
matplotlib-inline==0.1.3  # via ipython
parso==0.8.3              # via jedi
pexpect==4.8.0            # via ipython
pickleshare==0.7.5        # via ipython
plumbum==1.7.2            # via ptrender (pyproject.toml)
prompt-toolkit==3.0.29    # via ipython
ptyprocess==0.7.0         # via pexpect
pure-eval==0.2.2          # via stack-data
pygments==2.12.0          # via ipython
pyratemp==0.3.2           # via ptrender (pyproject.toml)
python-dateutil==2.8.2    # via strictyaml
requests==2.27.1          # via flit
six==1.16.0               # via python-dateutil
stack-data==0.2.0         # via ipython
strictyaml==1.6.1         # via ptrender (pyproject.toml)
tomli==2.0.1              # via flit
tomli-w==1.0.0            # via flit
traitlets==5.2.1.post0    # via ipython, matplotlib-inline
urllib3==1.26.9           # via requests
wcwidth==0.2.5            # via prompt-toolkit

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Is this because the flit backend implements the standard, and poetry doesn't? If so, is it just a matter implementing it in poetry?

[thisstillwill]

Any updates on this? With pyproject.toml I've been able to move most of my project configs for pytest, isort, etc. into a single file. It'd be great to do the same for the dependencies I have in requirements.in right now

[ssbarnea]

I migrated like 10+ projects to PEP-621 and using pip-tools with it, not sure what else is needed on that aspect. Maybe the only missing bit is that pip-tools does not include pyproject.toml among the implicit input files. I was planning to fix that, but once does is done we can fully close this.

Shortly, already done, is just a small tuning that could make ease easier, nothing more.

[AndydeCleyre]

Can anyone provide a PEP 621 sample that pip-tools doesn't currently handle?

[AndydeCleyre]

@jancespivo @thisstillwill

This issue doesn't yet have enough information to do anything with, namely a sample input and output that is not working as desired. If no one can supply that, I'm inclined to close this as in-actionable. Can you offer any more info?

[jancespivo]

@AndydeCleyre I've checked it and it works already, probably fixed in different issue.

pyproject.toml:

name = "some-package"
version = "0.0.0.dev"
requires-python = ">=3.8"

dependencies = [
    "kubernetes",
    "pyyaml",
    "pydantic",
]

pip-compile

Output:

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile
#
annotated-types==0.6.0
    # via pydantic
cachetools==5.3.2
    # via google-auth
certifi==2023.7.22
    # via
    #   kubernetes
    #   requests
charset-normalizer==3.3.2
    # via requests
...

From my point of view it is ok to close this issue. Don't know what others have to say.

Thanks!