Extra marker is ignored in poetry install, but respected in pip install

[Omegastick]

• Poetry version: 1.4.1
• Python version: 3.9
• OS version and name: Ubuntu 22.04 on WSL2
• pyproject.toml: https://gist.github.com/Omegastick/713164402b450b16b35c84f425db0a29

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

If my understanding is correct, a dependency like this should install the PyPI repository version of the dependency if installed as package_a[repo], and the path dependency if installed as package_a{local]:

[tool.poetry.dependencies]
package-b = [
    {version = "^0.1.0", markers = "extra == 'repo'", optional = true},
    {path = "../package_b", develop = true, markers = "extra == 'local'", optional = true},
]

[tool.poetry.extras]
repo = ["package-b"]
local = ["package-b"]

pip install -e .[local] installs the path dependency, and pip install -e .[repo] fails because package_b doesn't exist (as it should). poetry install -E local and poetry install -E repo both install the path dependency, though.

Use case

I have quite a few repos, all using Poetry, that I would like to install locally with a single poetry install at the top of the dependency tree.

Example dependency tree:

package_a -> package_b -> package_c

If you alter the package_b dependency specification to add the right extras:

package-b = [
    {version = "^0.1.0", extras = ["repo"], markers = "extra == 'repo'", optional = true},
    {path = "../package_b", extras = ["local"], develop = true, markers = "extra == 'local'", optional = true},
]

You could use poetry install -E local to install the whole tree for local development. pip install -e .[local] seems to work this way, unless I'm missing something.

I'm aware that including path dependencies outside of groups specifies the full path (not relative) in PKG-INFO. That's not a problem for us, because the packages are only being uploaded to a private PyPI.

[dimbleby]

you are stretching poetry quite far here, I dunno whether I'd expect this to work or not.

But I'd try an explicit source = "pypi" on the repo version of the dependency, so that poetry knows that requirement cannot be satisfied by the path version.

[Omegastick]

package-b = [
    {version = "^0.1.0", source = "pypi", markers = "extra == 'repo'", optional = true},
    {path = "../package_b", develop = true, markers = "extra == 'local'", optional = true},
]

This doesn't seem to work either. Interestingly, even with explicitly specifying the source, only the path dependency makes it into poetry.lock.

[[package]]
name = "package-b"
version = "0.1.0"
description = ""
category = "main"
optional = true
python-versions = "^3.9"
files = []
develop = true

[package.source]
type = "directory"
url = "../package_b"

[extras]
local = ["package-b", "package-b"]
repo = ["package-b", "package-b"]

[dimbleby]

probably duplicate #7722 then, there's a suggested fix in there if you want to run with it

[Omegastick]

Thanks for the help. It could be the same underlying issue. I installed Poetry from source and tried the suggested fix, but the code path doesn't seem to be hitting that method. I ran it in pdb and put a breakpoint there to be sure.

After looking at that issue, I tried switching the order of the dependencies. Putting the path dependency first, and the PyPI dependency second raises the Because package-a depends on package-b (^0.1.0) which doesn't match any versions, version solving failed. error, implying that it's the second package (rather than the first as in #7722) that is being installed.

I tried swapping out package_b for requests to confirm with a package that actually exists on PyPI, but that doesn't show the same behaviour. This dependency always installs the path dependency, regardless of the order:

requests = [
    {version = "^2.28.2", source = "pypi", markers = "extra == 'repo'", optional = true},
    {path = "../requests", develop = true, markers = "extra == 'local'", optional = true},
]

I'm happy to help with a fix, but I don't really know enough about Poetry's internals to suggest one.

[Omegastick]

Did some digging on this on this over the weekend:

Changing src/poetry/puzzle/provider.py:687 from
pep_508_dep = dep.to_pep_508(False)
to
pep_508_dep = dep.to_pep_508(True)
correctly calculates the overrides needed to specify both the local and repo versions of the requirement in poetry.lock. It was explicitly ignoring the extra marker. I'm sure making it respect the extra marker here has some knock-on effects, but I'm not sure what.

There's weird behaviour at install time, though. No -E flag correctly doesn't install anything, but passing either -E local or -E repo results in alternating between the local and repo versions of the dependency each install. So if you have the repo version installed, the local version will be installed, and vice versa. I'll have another look into this at some point.

[rafaela00castro]

It would be nice to have this feature working in poetry because it's extremely useful to have a separate version depending on the environment and be able to control that with --extras.

[marhoy]

It would be nice to have this feature working in poetry because it's extremely useful to have a separate version depending on the environment and be able to control that with --extras.

Agreed! I'm facing a similar issue where I want to install torch with either GPU- or CPU-support. I want to be able to control which one to choose with either a group or with extras.

[david-waterworth]

For me using

[tool.poetry.extras]
cpu = ["torch"]
cu117 = ["torch"]

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    {version = "2.0.0", markers = "extra == 'cpu' and extra != 'cu117'", source = "torch+cpu", optional=true},
    {version = "2.0.0", markers = "extra == 'cu117' and extra != 'cpu'", source = "torch+cu117", optional=true}
    ]

Results in poetry (1.7.1) trying to install both version (i.e. poetry install -E cu117

  •Installing torch (2.0.0+cpu)
  •Installing torch (2.0.0+cu117)

And then next time

  • Installing nvidia-cusolver-cu12 (11.4.5.107)
  • Installing nvidia-nccl-cu12 (2.19.3)
  • Installing nvidia-nvtx-cu12 (12.1.105)
  • Updating triton (2.0.0 -> 2.2.0)
  • Downgrading torch (2.0.0+cu117 -> 2.0.0+cpu)
  • Updating torch (2.0.0+cu117 -> 2.2.0)

Note first time it was installing torch+cuda but it didn't install the cuda (nvidia*) packages. Second time it installed the wrong nvidia packages (from torch 2.2.0)

Also I'd prefer to be able to add torch to optional groups, each with different versions because my project generates AWS Sagemaker ML Pipelines and I need to build/test containers - each with it's own requirements - so I'm using/abusing poetry to create separate environments for each step (in the container). My build script itself only needs sagemaker/boto3 but my tests/container Dockerfiles install specific groups from my poetry file. I know this pushes poetry beyond it's intended use.

[david-waterworth]

So part of what's happening here is I have 3 sources, our private codeartifact repo (which has PyPI as upstream) and two pytorch sources (CPU and GPU).

When I added transformers[torch] it's resolving the transformers extra (torch) against private rather than the pytorch sources so there's a conflict (or at least there's a newer version of torch available from private that satisfies the transfomers requirements so it takes precedence).

So I've replaced transformers[torch] with transformers and given up trying to use markers to select cpu/gpu for now.

[QuentinSoubeyranAqemia]

IMHO fixing this would provide the last missing piece to be able to use PyTorch and Poetry in all situations, using the setup outline in this earlier comment.

Poetry currently supports a single hardware variant of pytorch, but not switching between hardware variants, due this issue. This would be an awesome capability.

[GOGKI]

Hello Everyone, Is there a progress on this issue?