New python manager for PEP621

[wwuck]

What would you like Renovate to be able to do?

Renovate has existing managers for various python sources. The python community has recently accepted PEP621 as a standard format for providing python package metadata in a pyproject.toml file so that multiple packaging-related tools can consume this data from one location instead of various tool-specific configurations. This metadata includes dependencies/optional-dependencies.

I would like to request that Renovate gains a new manager for PEP621/pyproject.toml to maintain and keep updated the dependencies and optional_dependencies fields.

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[wwuck]

New package manager questionnaire

Did you read our documentation on adding a package manager?

• I've read the adding a package manager documentation.

Basics

Name of package manager

PEP621

What language does this support?

Python

How popular is this package manager?

Relatively new but gaining popularity among the popular python build tools. PEP621 has already been implemented by flit, pdm, trampolim, whey, enscons. PEP621 is also planned for inclusion in setuptools and is under consideration for poetry as well.

Does this language have other (competing?) package managers?

• Yes (give names)
• No

PEP621 is not a "package manager" itself, but a common format for defining static metadata that other python build tools/package managers can consume.
E.g. if PEP621 is implemented in setuptools it would be "competing with" and most likely superseding setup.py and setup.cfg for specifying the core python project metadata (which includes the package dependency data that Renovate would care about).

---

Package File Detection

What type of package files and names does it use?

pyproject.toml

What fileMatch pattern(s) should be used?

This would be similar to the existing poetry package manager in Renovate as they both use pyproject.toml, but PEP621 doesn't use a lock file such as poetry.lock.

Is it likely that many users would need to extend this pattern for custom file names?

• Yes
• No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No. There is only one file, pyproject.toml, associated with PEP621.

---

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No, there is only a single pyproject.toml file.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

What format/syntax is the package file in?

• JSON
• TOML
• YAML
• Custom (explain below)

How do you suggest parsing the file?

• Off the shelf parser
• Using regex
• Custom-parsed line by line
• Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

• Yes, required dependencies and optional dependencies
• No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

https://www.python.org/dev/peps/pep-0621/#dependencies-optional-dependencies
https://www.python.org/dev/peps/pep-0621/#example

Describe which types of dependencies above are supported and which will be implemented in future

---

Versioning

What versioning scheme does the package file(s) use?

Dependencies are specified according to PEP-508, which itself says that dependency versions are specified according to PEP440.

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

• Supports range constraints (e.g ^1.0.0 or 1.x)
• No

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

• Applications
• Libraries
• Both (explain how to tell which is which)

As I understand it, PEP621 is not concerned with application vs library. That would be the job of the build tool/package manager that is consuming the PEP621 metadata.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

---

Lookup

Is a new datasource required? Provide details

• Yes, provide details.
• No.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Yes, this would be similar to other python package managers implemented by Renovate such as setup.py and setup.cfg. This information is not considered part of the PEP621 spec.

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Yes. Only python3 would be supported as python2 has been EOL as of 2020-01-01.

Will users need the ability to configure language or other constraints using Renovate config?

---

Artifacts

Are lock files or checksum files used? Are they mandatory?

I think this is be outside the scope of PEP621, in the domain of the build tools/package managers that consume PEP621 metadata. I guess package versions would be specified directly in the relevant section of pyproject.toml?

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

I think this is be outside the scope of PEP621, in the domain of the build tools/package managers that consume PEP621 metadata. E.g. pip, poetry, pipenv.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Other

Is there anything else to know about this package manager?

[HonkingGoose]

@wwuck Thank you for using the issue template, and filling out the questionnaire! 😄

[HonkingGoose]

@rarkins It seems we still need a minimal reproduction repository? What do you need to proceed?

[rarkins]

Yes a repo or more with example use cases would help.

Also, once different tools implement lock files then we may need to rethink how to handle it.

[pradyunsg]

Here's some repositories with PEP 621 style dependency declarations:

• https://github.com/pypa/flit/blob/main/pyproject.toml

Notice that all of these declarations are "abstract" (i.e. not exact versions/artifacts, but rather a description of what version ranges are compatible) and not "concrete" (like a lock file that pins all artifacts and hashes would be).

[nuztalgia]

Adding a +1 to this request, as I just discovered pyproject.toml wasn't supported by Renovate after migrating to it in one of my repos. 😅 Based on the comments/tags in this issue, it looks like more examples/repros are necessary before it can be considered, so here's some info about my specific user experience with searching for a modern Python dependency manager that's supported by Renovate. I've included snapshots of the relevant dependency specification files and log excerpts from the corresponding Renovate jobs for each manager that I tried out.

---

Pipfile

The old faithful, and happily supported by Renovate. This is where I originally put my dependencies, before deciding that I wanted to "modernize" my project.

https://github.com/nuztalgia/qibot/blob/566d9c91b53f79b75a135866df40aa21dce56ebe/Pipfile#L6-L10

Log excerpts from job ID #730618240 (click to expand)

DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "566d9c91b53f79b75a135866df40aa21dce56ebe",
[...]
DEBUG: Matched 1 file(s) for manager pipenv: Pipfile
DEBUG: pre-commit is disabled
DEBUG: pipenv.extractPackageFile()
DEBUG: Found pipenv package files
DEBUG: Found 1 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {
      "pipenv": {
        "fileCount": 1,
        "depCount": 4
      }
    },
    "total": {
      "fileCount": 1,
      "depCount": 4
    }
  }
}

pyproject.toml

I did some research on current Python packaging practices, and per PEP 621, I decided this was the ideal place to specify my dependencies... until I realized it wasn't supported by Renovate.

https://github.com/nuztalgia/qibot/blob/7add85d229559f5caf93f6f018d984098b59ece2/pyproject.toml#L20-L32

Log excerpts from job ID #732082846 (click to expand)

DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "7add85d229559f5caf93f6f018d984098b59ece2",
[...]
DEBUG: Matched 1 file(s) for manager pip_setup: setup.py
DEBUG: Matched 1 file(s) for manager poetry: pyproject.toml
DEBUG: pre-commit is disabled
DEBUG: pyproject.toml contains no poetry section
DEBUG: Found 0 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {},
    "total": {
      "fileCount": 0,
      "depCount": 0
    }
  }
}

setup.py

This is where I've put my dependencies for the time being, in order to maintain compatibility with Renovate - although I have to admit, this workaround feels quite unintuitive/undesirable.

https://github.com/nuztalgia/qibot/blob/59b8e6d7a70d7f6c5d5fef7c0526d0b8fafa4c7b/setup.py#L4-L16

Log excerpts from job ID #732754677 (click to expand)

DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "59b8e6d7a70d7f6c5d5fef7c0526d0b8fafa4c7b",
[...]
DEBUG: Matched 1 file(s) for manager pip_setup: setup.py
DEBUG: Matched 1 file(s) for manager poetry: pyproject.toml
DEBUG: pre-commit is disabled
DEBUG: pyproject.toml contains no poetry section
DEBUG: Found pip_setup package files
DEBUG: Found 1 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {
      "pip_setup": {
        "fileCount": 1,
        "depCount": 7
      }
    },
    "total": {
      "fileCount": 1,
      "depCount": 7
    }
  }
}

---

Hopefully this helps provide some more context to support this issue. Let me know if any additional information would be useful!

[mkniewallner]

Handling PEP 621 may actually be a bit complex to handle.

PEP 621 (and by extension, PEP 631) defines the structure for declaring dependencies, which is convenient since projects implementing the specification share a common structure.

But I believe that from a Renovate perspective, this would not really be a manager per se, but rather a common reusable structure and logic to parse, which itself is used by different managers (for instance, for libraries following PEP 621: PDM, setuptools, flit, hatch, ...).

Despite following the same structure for specifying dependencies, tools implementing PEP 621 still have their own lock mechanism (some also don't handle locking) for the lock file (both for the format and the name), since this is not covered by a PEP.

Also, since Renovate would need to run commands from the different libraries, which will most likely differ, there definitely needs to be a differentiation between the different tools.

So depending on the tool used by projects, Renovate would need a hint on the tool that was used. Something naive may be to look for a specific lock file name (since the tools will most likely have their own file name, which have a great chance of being unique).

In the future, if there's ever an accepted common format for the lock file (there was PEP 665, but it has been rejected), this may become unreliable, though, but maybe this is looking too much into what the future may become.

[flying-sheep]

I think you’re almost spot on, except for the “unreliable” part, as the future could look like this:

1. we add support the following managers:

   • PEP 621’s pyproject.toml + pdm.lock
   • PEP 621’s pyproject.toml + pip-tools’ requirements.txt

2. Once a standard lock file exists we support an additional manager:

   • PEP 621’s pyproject.toml + <standard lockfile>

I don’t see how this could become unreliable or otherwise prevent action right now.

[ofek]

Please keep in mind that libraries do not need lock files and will only ever use pyproject.toml. Also, this should imo be called standard.

[djmattyg007]

Please keep in mind that libraries do not need lock files

I don't think this is accurate at all. Lock files typically speed up the installation process drastically, which means they're a good idea to commit in all places. It also ensures you're using the exact same version for testing things in all environments. Libraries still have plenty of dev dependencies, and when a team is working on that one library, it's important that everyone is using the same version of those dev dependencies.

[ofek]

Generally, I think it's irresponsible to use lock files for libraries rather than pulling latest to catch incompatibilities before end users do.

Lock files typically speed up the installation process drastically

Dependency resolution pales in comparison to the downloading and unpacking of artifacts, unless that is in fact necessary for dependency resolution as is the case for Python source distributions when no wheels are available.

Though this is not very applicable to libraries as there won't be potentially hundreds or thousands of dependencies like applications may have.

Libraries still have plenty of dev dependencies, and when a team is working on that one library, it's important that everyone is using the same version of those dev dependencies.

Those can just be pinned since most libraries do the approach I described and will catch issues with the latest versions of their transitive dependencies before you do 😉

[pradyunsg]

puts on the hat of "I am one of the authors of pip's dependency resolver"

@ofek is correct IMO - if libraries shipped lock files to their end users, that would be inappropriate and would be equivalent to overly aggressive pinning.

There might be a case to be made for development workflows but as @ofek noted, you typically want to get the latest versions to catch issues before end-users see them. Or, at least, end users who are consuming it via a lock file and being judicious about when/how they update things.

PEP 621’s pyproject.toml + pip-tools’ requirements.txt

I don't know much about renovate's semantics/specifics but I'll note that pip-tools already supports this workflow.

[djmattyg007]

if libraries shipped lock files to their end users, that would be inappropriate and would be equivalent to overly aggressive pinning

This simply isn’t true. Lock files are not used by dependent software. The contents of a library’s lock file has no bearing on anything outside of that project itself.

[dvzrv]

Hi! As a user of PDM for some projects, I'm wondering whether anyone is working on implementing this for renovate (as described in #10187 (comment)).

What would be needed to make this happen?

[viceice]

Please don't post me too comments, they are too noisy. We'll delete them in future.

To get this happen somebody from the community needs to send a PR with an implementation.

it also needs a reproduction as the label states.

[renovate-release]

🎉 This issue has been resolved in version 35.83.0 🎉

The release is available on:

• GitHub release
• 35.83.0

Your semantic-release bot 📦🚀

[secustor]

Thanks to Vector Informatik GmbH for sponsoring this feature!