Ensure optional dev-dependencies are written to extras.

[nickpresta]

Problem

When specifying an extras target that contains packages from dev-dependencies, those packages were uninstallable.

Example

pyproject.toml

[tool.poetry]
name = "mything"
version = "0.1.0"
description = "description"
authors = ["nick"]

[tool.poetry.dependencies]
python = "^3.6 || ^3.7"

[tool.poetry.dev-dependencies]
pytest = "^3"
black = { version = "^18.9b0", optional = true }

[tool.poetry.extras]
developer = ["black"]

Running poetry install -v results in the following lockfile. The important bit is that the extras section lists:

[extras]
developer = []

Which is wrong (we should see black in there).

poetry.lock

[[package]]
category = "dev"
description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
name = "appdirs"
optional = true
python-versions = "*"
version = "1.4.3"

[[package]]
category = "dev"
description = "Atomic file writes."
name = "atomicwrites"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.2.1"

[[package]]
category = "dev"
description = "Classes Without Boilerplate"
name = "attrs"
optional = false
python-versions = "*"
version = "18.2.0"

[[package]]
category = "dev"
description = "The uncompromising code formatter."
name = "black"
optional = true
python-versions = ">=3.6"
version = "18.9b0"

[package.dependencies]
appdirs = "*"
attrs = ">=17.4.0"
click = ">=6.5"
toml = ">=0.9.4"

[[package]]
category = "dev"
description = "Composable command line interface toolkit"
name = "click"
optional = true
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "7.0"

[[package]]
category = "dev"
description = "Cross-platform colored terminal text."
marker = "sys_platform == \"win32\""
name = "colorama"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "0.4.0"

[[package]]
category = "dev"
description = "More routines for operating on iterables, beyond itertools"
name = "more-itertools"
optional = false
python-versions = "*"
version = "4.3.0"

[package.dependencies]
six = ">=1.0.0,<2.0.0"

[[package]]
category = "dev"
description = "plugin and hook calling mechanisms for python"
name = "pluggy"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "0.8.0"

[[package]]
category = "dev"
description = "library with cross-python path, ini-parsing, io, code, log facilities"
name = "py"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.7.0"

[[package]]
category = "dev"
description = "pytest: simple powerful testing with Python"
name = "pytest"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "3.10.0"

[package.dependencies]
atomicwrites = ">=1.0"
attrs = ">=17.4.0"
colorama = "*"
more-itertools = ">=4.0.0"
pluggy = ">=0.7"
py = ">=1.5.0"
setuptools = "*"
six = ">=1.10.0"

[[package]]
category = "dev"
description = "Python 2 and 3 compatibility utilities"
name = "six"
optional = false
python-versions = "*"
version = "1.11.0"

[[package]]
category = "dev"
description = "Python Library for Tom's Obvious, Minimal Language"
name = "toml"
optional = true
python-versions = "*"
version = "0.10.0"

[extras]
developer = []

[metadata]
content-hash = "51e9a7833e4507f665108d8c37b912e5533b82885eaeeeb65751f0efdf0254cd"
python-versions = "^3.6 || ^3.7"

[metadata.hashes]
appdirs = ["9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92", "d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"]
atomicwrites = ["0312ad34fcad8fac3704d441f7b317e50af620823353ec657a53e981f92920c0", "ec9ae8adaae229e4f8446952d204a3e4b5fdd2d099f9be3aaf556120135fb3ee"]
attrs = ["10cbf6e27dbce8c30807caf056c8eb50917e0eaafe86347671b57254006c3e69", "ca4be454458f9dec299268d472aaa5a11f67a4ff70093396e1ceae9c76cf4bbb"]
black = ["817243426042db1d36617910df579a54f1afd659adb96fc5032fcf4b36209739", "e030a9a28f542debc08acceb273f228ac422798e5215ba2a791a6ddeaaca22a5"]
click = ["2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13", "5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7"]
colorama = ["a3d89af5db9e9806a779a50296b5fdb466e281147c2c235e8225ecc6dbf7bbf3", "c9b54bebe91a6a803e0772c8561d53f2926bfeb17cd141fbabcb08424086595c"]
more-itertools = ["c187a73da93e7a8acc0001572aebc7e3c69daf7bf6881a2cea10650bd4420092", "c476b5d3a34e12d40130bc2f935028b5f636df8f372dc2c1c01dc19681b2039e", "fcbfeaea0be121980e15bc97b3817b5202ca73d0eae185b4550cbfce2a3ebb3d"]
pluggy = ["447ba94990e8014ee25ec853339faf7b0fc8050cdc3289d4d71f7f410fb90095", "bde19360a8ec4dfd8a20dcb811780a30998101f078fc7ded6162f0076f50508f"]
py = ["bf92637198836372b520efcba9e020c330123be8ce527e535d185ed4b6f45694", "e76826342cefe3c3d5f7e8ee4316b80d1dd8a300781612ddbc765c17ba25a6c6"]
pytest = ["630ff1dbe04f469ee78faa5660f712e58b953da7df22ea5d828c9012e134da43", "a2b5232735dd0b736cbea9c0f09e5070d78fcaba2823a4f6f09d9a81bd19415c"]
six = ["70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9", "832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"]
toml = ["229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c", "235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e", "f1db651f9657708513243e61e6cc67d101a39bad662eaa9b5546f789338e07a3"]

Running poetry install -E developer results in:

$ poetry install -E developer
Installing dependencies from lock file

Nothing to install or update

which makes sense (no packages in the developer target).

Potential Workaround

By moving the black package out of dev-dependencies and into dependencies, the developer target correctly includes black.

This is not ideal -- my black, mypy, etc dependencies are "dev" dependencies and moving them to dependencies is "wrong" but also confuses the installation requirements.

Solution

We want to ensure that all requirements, whether from dependencies or dev-dependencies for extras targets are written to the lockfile.

This should fix #426 and potentially other, related issues.

I first wrote a failing test and then updated the implementation to make the test pass:

Failing Test

$ poetry run pytest -k test_poetry_writes_extras_from_dev_dependencies -s
Test session starts (platform: darwin, Python 3.7.0, pytest 3.9.1, pytest-sugar 0.9.1)
rootdir: /Users/npresta/poetry, inifile:
plugins: sugar-0.9.1, mock-1.10.0, cov-2.6.0


――――――――――――――――――――――――――――――――――――――――――――――――――― test_poetry_writes_extras_from_dev_dependencies ―――――――――――――――――――――――――――――――――――――――――――――――――――

    def test_poetry_writes_extras_from_dev_dependencies():
        poetry = Poetry.create(str(fixtures_dir / "project_with_extras"))

        package = poetry.package

        dev_dependencies = {}
        for dep in package.dev_requires:
            dev_dependencies[dep.name] = dep

        black = dev_dependencies["black"]
        assert black.is_optional()

        assert "extras_c" in package.extras
        extras_c = package.extras["extras_c"]
>       assert len(extras_c) == 1
E       assert 0 == 1
E        +  where 0 = len([])

tests/test_poetry.py:174: AssertionError

 tests/test_poetry.py ⨯                                                                                                                 100% ██████████
================================================================== warnings summary ===================================================================
/Users/npresta/Library/Caches/pypoetry/virtualenvs/poetry-py3.7/lib/python3.7/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Results (1.56s):
       1 failed
         - tests/test_poetry.py:160 test_poetry_writes_extras_from_dev_dependencies
     421 deselected

Passing Test

$ poetry run pytest -k test_poetry_writes_extras_from_dev_dependencies -s
Test session starts (platform: darwin, Python 3.7.0, pytest 3.9.1, pytest-sugar 0.9.1)
rootdir: /Users/npresta/poetry, inifile:
plugins: sugar-0.9.1, mock-1.10.0, cov-2.6.0

 tests/test_poetry.py ✓                                                                                                                 100% ██████████
================================================================== warnings summary ===================================================================
/Users/npresta/Library/Caches/pypoetry/virtualenvs/poetry-py3.7/lib/python3.7/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Results (1.35s):
       1 passed
     421 deselected

Checklist

I didn't update documentation because this is how the feature should have always worked, IMO, and thus it was a bug. Once this is merged in, the existing documentation should suffice.

• Added tests for changed code.
• Updated documentation for changed code.

[sdispater]

Thanks for taking the time to make this PR!

However, this is not something I want to add. Extras will be referenced in the distributions metadata when packaging the project but development dependencies do not which will lead to a broken extras.

[nickpresta]

@sdispater thanks for the response. Is the thinking that you would specify the packages in dependencies but always make them optional? That way, consumers who did not specify the extras won't be required to install the packages, but local development on the package can install the extras manually?

[revmischa]

This is a confusing problem to track down. Maybe warn if something is dev and optional, because there's no way to install it?

[fgimian]

I hit this issue today, and imho the current behaviour is very confusing 😢

[rafrafek]

Dev dependencies cannot be optional, but not marking them as optional causes another issue: python-poetry/poetry-plugin-export#34 (comment)

Is there any workaround?

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.