Feature Proposal: PEP 735 "Dependency Groups" Support

[sirosen]

What's the problem this feature will solve?

"Dependency Groups" are proposed in PEP 735 with a variety of target use cases.

In particular, the --only-deps use cases are explicitly a target for this PEP, but additional cases are listed and relevant.

Describe the solution you'd like

As the PEP author, I would like to start discussing what form, in terms of interface and implementation, pip support would take.
I would like to agree on a potential implementation path, implement it in a draft PR, and leave it pending until the PEP is accepted.

My proposal is:

• a variant of the DependencyGroupResolver I wrote moves in-tree here, along with appropriate unit tests
• a new CLI option is added to pip install, --dependency-groups, which takes a comma-delimited list of group names and attempts to resolve and install all of them from pyproject.toml in the current working directory

For example,

# in pyproject.toml
[dependency-groups]
test = ["pytest"]
typing = ["pyright"]

# on the CLI
pip install --dependency-groups test,typing

# equivalent to
pip install pytest pyright

Would the pip maintainers be amenable to this addition? Is there an alternative, similar strategy which I could implement?

Alternative Solutions

If pip does not gain support for Dependency Groups, it is possible to workaround the gap.
Primarily, external tooling like dependency-groups can be used to resolve groups and pass them to pip.

For example, in a *nix environment, the following usage covers most cases:

groups=$(python -m dependency_groups pytest typing | tr '\n' ' ')
pip install $groups

However, such usages will always be workarounds. pip support provides a superior developer experience.

Additional context

PEP 735 is in Draft, and will be submitted soon. At time of writing, it is not final or accepted, but no further changes to the contents are planned.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Some questions around the UI.

How would something like pip install requests --dependency-groups xxx work? Is the flag intended to only be valid for installing from source? What about pip install requests --no-binary requests --dependency-groups xxx in that case?

How will the flag work with multiple requirements? If I do pip install --dependency-groups foo ./a ./b ./c is the dependency group looked up for all of a, b, and c? Or is this invalid usage?

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

[RonnyPfannschmidt]

I'd consider pip the wrong place for managing this in a normal way

IMHO it's different enough from normal install that it warrants a distinct command to install dependency groups of a specific source folder

Tools like hatch, rye and poetry should be the place of closer integration

[sirosen]

I am a bit concerned about meeting the target use cases for the PEP without pip support. It would be possible, but the user experience is worse in several cases.

IMHO it's different enough from normal install that it warrants a distinct command to install dependency groups of a specific source folder

That may be, but I see the feature as similar to -r. The main difference being that the path lookup is implicit in the UX I proposed -- it needn't be. We can discuss usages in which the path to pyproject.toml is given explicitly. (More on this below)

Tools like hatch, rye and poetry should be the place of closer integration

I think the answer here is "and" not "or". Dependency Groups will have the greatest positive impact if they are supported by installers, environment managers, and build backends.

I've started with pip and setuptools. Perhaps this is a strategy mistake and I should begin from younger tools like flit.

---

Before I dive into Paul's specific scenarios, I want to start from the high level concern, make sure I understand it, and consider whether or not the UX I proposed is correct.

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

Is the issue that this is expected to expand to a set of requirements, whereas -r expands to a set of arguments? Or is it the implicit lookup of ./pyproject.toml? Both of these? Something else?

I'm trying to understand if the issue would be mitigated by making the toml file explicit, e.g., pip install --dependency-group pyproject.toml:foo --dependency-group ../otherproj/pyproject.toml:bar.

[RonnyPfannschmidt]

I believe the proposed installation ux clearly shows my 3rd point

[pfmoore]

I think the problem for me was that it wasn't clear that you're proposing that --dependency-groups works the same way as --requirement. In particular, in the pip install help, we currently have

python -m pip install [options] <requirement specifier> [package-index-options] ...
python -m pip install [options] -r <requirements file> [package-index-options] ...
python -m pip install [options] [-e] <vcs project url> ...
python -m pip install [options] [-e] <local project path> ...
python -m pip install [options] <archive url/path> ...

If I understand, you're proposing to add a new form here:

python -m pip install [options] --dependency-groups <list of groups>  [package-index-options] ...

This would look in a pyproject.toml in the current directory and extract the list of requirements to install from the specified dependency groups in there. Looking at it that way seems a lot more plausible, and eliminates all of my objections above. We do allow -r and "normal" requirements to be combined, even though the syntax above doesn't show it. I'd assume --dependency-groups would work the same.

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

That was incorrect, we do, it's -r. If we mirror the behaviour of -r, that should be OK. Sorry for the confusion here.

There's some background information available in #8049. The conclusion there was that we rejected the feature request, but dependency groups are much more specific, and will be standards-based, so I think it's within scope for pip to support them.

Also, there may be some edge cases that won't work, because requirements files are much richer than dependency groups. For example, --dependency-groups won't work well with hash checking mode, as (unlike requirement files) dependency groups don't offer a way to specify a hash. I think that's fine, but it might be worth explicitly documenting any limitations we are aware of somewhere, so we don't get user reports saying "hash checking fails with dependency groups" and expecting us to fix this.

[pfmoore]

We can discuss usages in which the path to pyproject.toml is given explicitly.

Strong -1 on me for allowing anything but "read dependency groups from pyproject.toml in the current directory". Finding the pyproject.toml for some idea of the "current project" should be left to project management tools, and explicitly stating the path to pyproject.toml is unnecessary complexity that (as far as I'm aware) there's no valid use case for.

[notatallshaw]

Strong -1 on me for allowing anything but "read dependency groups from pyproject.toml in the current directory". Finding the pyproject.toml for some idea of the "current project" should be left to project management tools, and explicitly stating the path to pyproject.toml is unnecessary complexity that (as far as I'm aware) there's no valid use case for.

So one of the repos I work with is a messy monorepo, in that repo we would never put a pyproject.toml in there because it might stomp all over the settings of one team or another.

This has never been a problem because front end and utility tools, like pip, pre-commit, linters, formatters, etc. allow you to specify where you want the configuration / requirements to come from. And backend tools that do not allow this are generally not suitable for the repo anyway i.e. we never want to build a package in the root of the repo.

Were pip to use this as a "requirements-like install file but with structured options" it would be a departure from pip's current approach to allow the user to specify the path of the requirements file or constraints file, and other front end tools in general.

If you don't consider "messy monorepo" as a valid use case, fair enough, but we're not trying to build a package from it, so it would be annoying for frontend tools, that are not project management tools, to say how we should structure our project.

That said, this isn’t a hill I want to die on, just want to give an alternative view point.

[sirosen]

I think "messy monorepo" is a valid use-case to consider for pip (or, really, any ecosystem tool), but considering it isn't the same as letting it drive the UX/DX.

If pip support for dependency groups implicitly always used ./pyproject.toml, I can see that it might hamper some usages, but not that it would block them entirely. You can always cd ... (shell), WORKDIR ... (docker), or similar -- I can't think of any tools capable of running pip which are deficient in this regard.

I'm not strictly against something like --dependency-groups-pyproject=... with a default of pyproject.toml, but I trust the judgement of the pip maintainers about what is a good fit (or not) for the existing tool.

---

I'm not sure if I should open a draft PR or not at this stage? I can add a pip wrapper to dependency-groups, e.g. pip-dependency-groups install ...?

[redeboer]

Update: PEP 735 has been accepted 🎉
https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233/312

[potiuk]

Update: PEP 735 has been accepted 🎉 https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233/312

Fantastic. We have some good use cases for those in Airflow and once pip and other tools will start supporting those, we will definitely make a very good use of those.

[henryiii]

FWIW, uv has shipped dependency groups (as has tox). It would potentially be nice to mimic the CLI interface names here unless there's a reason not to.

[sirosen]

I refrained from taking any further action here because it wasn't clear if the pip maintainers had sufficient agreement that we should add it. I didn't want to push the issue too aggressively, but my approach may have been too hands off. I'll work this up in the form of a PR as soon as I'm able to devote the time, and can drive discussion to a clearer yes-or-no decision.

In the meantime, dependency-groups provides pip-install-dependency-groups as a trivial wrapper for simple use cases.

---

uv has defined this as --group. So matching CLIs would expect pip to implement this as

# analogous to `pip install -r testing.txt -r typing.txt`
pip install --group testing --group typing

I don't have particularly strong feelings about what the option name should be, so I would be inclined to follow their lead and use --group unless there's some objection.

[henryiii]

Is it --group or --only-group? I believe --only-group is the one analogous to -r requirements.txt, with --group being analogous to -e. -r requirements.txt. Though maybe one could argue that the -e. part is implied in other commands as well, so pip should just be --group. pip install -e. --group dev looks pretty good.

[sirosen]

I think that using --only-group would be over-indexing on the analogy with uv. Since it's a workflow tool, having a notion of the current project and pyproject.toml is already baked into the model, so it needs disambiguation.

That said, maybe this is an argument against having the same UI, since users will incorrectly infer that the two do the same thing. I can just imagine the naive user tickets asking why pip install --group foo didn't install the "current project".

[notatallshaw]

Well, uv have added their to their project management CLI:

$ uv add --help | grep group
      --dev                          Add the requirements to the development dependency group
      --group <GROUP>                Add the requirements to the specified dependency group

But they have not added it to their pip CLI:

$ uv pip install --help | grep group
$

Which I think is a little telling... Or maybe they're waiting for a user request or see what pip does.

I don't see any mention of --only-group, I don't know where that comes from.

[alex]

They've said that they're waiting to see what pip does before adding anything to uv pip, for the obvious reasons :-)

[henryiii]

--only-group is listed in astral-sh/uv#8272.

I like --group the best, it's short and authoritative. And I think due to the fact the workflow commands all install -e. anyway, I don't think it would be too much of a clash. (And maybe the removed "install the project" part of the PEP can be resubmitted someday? I think it's a step backward to not have a way to require the project installed with the testing dependencies)

[henryiii]

Are you running an older version? I see it:

$ uvx [email protected] sync --help | grep group                                                                                     Fri Oct 25 16:52:04 2024
      --no-dev                                   Omit the development dependency group
      --only-dev                                 Only include the development dependency group
      --group <GROUP>                            Include dependencies from the specified dependency group
      --no-group <NO_GROUP>                      Exclude dependencies from the specified dependency group
      --only-group <ONLY_GROUP>                  Only include dependencies from the specified dependency group

(I'm one version behind, waiting on homebrew, so using uvx to check it)

Ah, you are using uv add, --only-group is not relevant when adding things to a group. Only when installing a group.

[sirosen]

And I think due to the fact the workflow commands all install -e. anyway, I don't think it would be too much of a clash.

Ah, good point. I think that mitigates my concern about supporting it in pip and users bringing an expectation that it will exactly match uv.

I'll work with --group for now, but I'll make a note at review-time that this is chosen to match uv and may need more thought.

(And maybe the removed "install the project" part of the PEP can be resubmitted someday? I think it's a step backward to not have a way to require the project installed with the testing dependencies)

Yes, once the dust settles, I do want to pursue this idea. (Or related ideas which cover this functionality.)
I'm not sure how far into the future that will be though.

[notatallshaw]

Ah, you are using uv add, --only-group is not relevant when adding things to a group. Only when installing a group.

Ah yes, I see they're not exposed via uv sync, but uv add also installs a group, and to my eyes add looks closer to pip install than sync.

Probably a cue that project semantics are not the same as install semantics and pip should choose independently of uv.

[zanieb]

We have some additional experience that may help here.

uv already supports uv pip install --extra. However, this is only allowed if you provide -r with a project source, e.g.:

❯ uv pip install --extra test anyio
error: Requesting extras requires a `pyproject.toml`, `setup.cfg`, or `setup.py` file.
❯ uv init example
❯ uv pip install -r example/pyproject.toml --extra test anyio
error: Requested extra not found: test
❯ uv pip install -e example --extra test anyio
error: Requesting extras requires a `pyproject.toml`, `setup.cfg`, or `setup.py` file.

In this case, we actually don't install the project itself, just the dependencies declared in the file. Generally, this is confusing to people using pip install, e.g.:

• Editable installation with extras astral-sh/uv#7845
• pyproject validation messages are ambiguous astral-sh/uv#8416
• FR: add support for uv pip install <PyPI package> --all-extras astral-sh/uv#7295
• uv pip install --all-extras . fail to locate pyproject.toml astral-sh/uv#4762
• uv pip compile does not include local package astral-sh/uv#6317

I believe we added this flag for uv pip compile use-cases (i.e., you want to export the requirements of your project), then propagated it throughout the interface. We needed a separate flag to power this because there are no semantics for requesting extras from a project file, e.g., pip install -e example[test] is standard but pip install -r pyproject.toml[test] is not.

Adding support for PEP 735 looks like a very similar problem. There is no standard way to request groups in the package specifier. In hindsight, this probably should have been considered in the PEP (I do not recall seeing discussion, if there was any, please point me to it!).

There's a bit more complexity here, in that pip does not support project files via -r like uv:

❯ pip install -r example/pyproject.toml
ERROR: Invalid requirement: '[tool.uv]': Expected package name at the start of dependency specifier
    [tool.uv]
    ^ (from line 1 of example/pyproject.toml)

So, where do we go from here? In uv, I think we'll need to support --group in uv pip for parity with --extra and the rest of our interface. However, there's not a clear equivalent for that flag here since there's not a way to provide a path to the pyproject.toml. Much of the conversation thus far has been focused on how a user says which project they want the groups to come from. Notably, I think "read dependency groups from pyproject.toml in the current directory" would be incredibly confusing; pip does not do any discovery of projects today, right? Does pip have any reliance on the contents of the working directory today? This seems like a weird and surprising change to the experience. I have a strong expectation that the path to the project would need to be provided.

Concretely, I'd recommend adding a --group flag to pip, which, enables the group for for any paths in the installation:

pip install <path> --group test

If there are no paths (e.g., pip install anyio --group test) or the group does not exist in any path, I'd expect pip to fail. I don't think groups should be read from source distributions. Groups should only be read from <path>/pyproject.toml, not discover a pyproject.toml elsewhere in the directory. Groups could be read from remote source directories like VCS URLs.

I think this is the most guessable solution for users, e.g., you'd just tack on pip install -e . --group dev. It also makes it clear the project itself is being included in the installation (which would otherwise be ambiguous), and gives control over installing the project in editable or regular mode.

This leaves an important use-case unaddressed: installing a group without the project itself. I'm not sure what the best path forward on this is for pip. uv pip install -r pyproject.toml --group dev is the most logical interface in uv, but we've seen significant user confusion around that interface and I can't strongly recommend it given there's not already support for -r pyproject.toml. Regardless I think you end up with some need to specify the project path via a dedicated flag, e.g., pip install --project . --group dev, or in the group specification (if you want to install groups from multiple projects), e.g., pip install --group .[dev] (similarly, proposed in #12963 (comment)). As a minor note, something like #11440 doesn't address this since the project's other dependencies would still be installed.

Similarly, we haven't figured out how to expose an interface for groups without a [project] table in uv, but the user expectation is that uv run will discover a pyproject.toml so something like uv run --group test should "just work". As I mentioned before, adding project discovery to pip (even in the working directory) feels like a big change. I think this is also a possible cause of behavior differences between uv and pip, as uv pip would probably need to follow our typical pyproject.toml workspace discovery rules which goes beyond the working directory.

To be clear, I don't have any expectation that pip conforms to uv's interface. I do think it's in the best interest of the users to have a coherent experience across tools though. I'm hoping that our experiences and feedback from users can help inform decisions here (and vice-versa!).

[pfmoore]

There is no standard way to request groups in the package specifier. In hindsight, this probably should have been considered in the PEP

I believe this was deliberate, and in any case it's correct IMO, as allowing requirement specifiers to include dependency groups would allow projects to depend on dependency groups, which isn't possible because dependency groups aren't stored in project metadata.

Requesting groups via an installer flag is the intended approach.

pip does not support project files via -r like uv

I don't think that's something we'd want to support, either. The --requirement (-r) flag is specifically to indicate a requirements file, it's not for specifying the pyproject.toml file.

This is the root of why dependency groups are a bit of an uncomfortable "fit" for pip. We very deliberately and explicitly don't require any sort of concept of "the current project" in order to use pip. But dependency groups are very much a project-oriented concept (because they are defined in the project metadata file!) But maybe that suggests a way forward here. We could add a new --project option to pip, which points to the "selected project" (a local directory). Note in particular that you specify the project rather than the pyproject.toml file. Any pip commands that need the concept of a project would need to specify it via the --project option - at the moment, that's just dependency groups, but longer term it might include other features based off the pyproject.toml file¹. So the way to install a dependency group would then be:

pip install --project . --group testing

Just to be clear, this is not a fully-formed idea at this point, it's just an off-the-cuff suggestion for an approach that fits better with pip's design. It will need some work to flesh it out.

Concretely, I'd recommend adding a --group flag to pip, which, enables the group for for any paths in the installation

In particular, I don't like this option. As you say, it doesn't offer a way to not install the project, and furthermore, it complicates things when there's multiple paths in the install command (which might not even agree on what group names mean). And it also makes local directory names special, in a way that would immediately prompt the question "why not allow reading pyproject.toml from sdists as well?" and then why not allow remote sdists (viat http)...

As I mentioned before, adding project discovery to pip (even in the working directory) feels like a big change.

Agreed, and at least personally, it's not a direction I want to go in (I haven't asked the other @pypa/pip-committers their views on this). In particular, while I don't want to fall into the trap of making every detail a big design question, I also don't want a significant change in philosophy to "slip in" as part of what should be a fairly straightforward feature addition.

To be clear, I don't have any expectation that pip conforms to uv's interface. I do think it's in the best interest of the users to have a coherent experience across tools though. I'm hoping that our experiences and feedback from users can help inform decisions here (and vice-versa!).

Thanks for that comment. And I do agree that being consistent is better for users. I think that pip and the uv pip interface should definitely try to be consistent, but we may have to diverge from the "native"² uv interface, simply because uv (in its native form) is a project focused tool, whereas pip isn't. I don't know how comfortable that would be for uv users, though (do your users expect uv pip to be more like pip, or more like uv?)

Footnotes

1. The proposed --only-deps option might fall into this category... ↩

2. Is there an official term for the uv interface that isn't uv pip? ↩

[zanieb]

I don't think that's something we'd want to support, either. The --requirement (-r) flag is specifically to indicate a requirements file, it's not for specifying the pyproject.toml file.

I agree. We adapted this from pip-tools then made it consistent across our interface — I don't think pip needs this.

Any pip commands that need the concept of a project would need to specify it via the --project option - at the moment, that's just dependency groups, but longer term it might include other features based off the pyproject.toml file1. So the way to install a dependency group would then be pip install --project . --group testing

Did you see I recommended this? Basically, yeah this makes sense to me. I especially like that you highlighted that there is future work that will require project awareness in pip. However, I worry that it'll be confusing for users to need a command like:

pip install -e . --project . --group test

This is what led me to having --group apply to source trees. Is there an easier way to install the project and the group? It seems worth optimizing for that over cases where you want the group separately from the project.

There are still confusing cases where it's not clear what group applies to, e.g.:

pip install ./bar --project ./foo foobar --group test

And it also makes local directory names special, in a way that would immediately prompt the question "why not allow reading pyproject.toml from sdists as well?" and then why not allow remote sdists (viat http)...

I think the idea here is that we would support this for source trees, but not distributions. I don't think it'd be hard distinction to make, i.e., we won't consume parts of the pyproject.toml that are not part of the distribution metadata. I understand your hesitation, but I feel like the entire focus of dependency groups is local. It seems feasible that users won't know that groups are even available in source distributions, and I think there's a lot of room to guide expectations with error messages. Most pip install invocations are probably for a single target package, which makes it less likely to be confusing (unless using a requirements file, in which case I don't think groups really applies). I think there will need to be some careful work to set expectations about what --group applies to regardless, since it's a different paradigm.

(do your users expect uv pip to be more like pip, or more like uv?)

They want both 🙃

In this context, it might be fine if we infer the value of --project even if pip does not since pip would just exit with an error otherwise.

Is there an official term for the uv interface that isn't uv pip?

I guess I'd say... "uv's interface" or "uv's project interface" vs "uv's pip-compatible interface". The project interface includes all of our top-level commands, though uv run can be used in more contexts.