poetry support for optional dependencies

[dazza-codes]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Existing behavior

See debug-versions below for details.

Assume this sample pyproject.toml

[tool.poetry]
name = "poetry-tmp-project"
version = "0.1.0"
description = ""
authors = ["''"]

[tool.poetry.dependencies]
python = "^3.6"
marshmallow = {version = "^3.7.1", optional = true}
pydantic = {version = "^1.6.1", optional = true}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Created by:

• poetry init # etc
• poetry add --optional marshmallow pydantic

Install optional packages?

Given the sample above, there are only optional packages and no extras that point to them. On a clean project, there is not yet any poetry.lock file.

$ poetry install
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

No dependencies to install or update

$ pip freeze
certifi==2020.6.20  # this is from the conda-env used to create a venv

$ poetry show -t
marshmallow 3.7.1 A lightweight library for converting complex datatypes to and from native Python datatypes.
pydantic 1.6.1 Data validation and settings management using python 3.6 type hinting
└── dataclasses >=0.6

• the poetry show command lists all the dependencies regardless of whether they are optional or not or whether or not they are installed; it might help to add an optional flag on it for something like --installed to only list the packages that are installed in the current venv

$ poetry install --optional

[NoSuchOptionException]
The "--optional" option does not exist.

$ poetry install --option marshmallow

[NoSuchOptionException]
The "--option" option does not exist.

• an --option or an --optional flag might be helpful when testing a package that requires all of the possible packages to be installed for purposes of unit-testing etc
• without any explicit definition of package extras for the optional packages, there seems to be no way to install them for purposes of unit-testing etc

Using optional extras

By default, optional packages cannot be installed as extras, e.g.

$ poetry install --extras marshmallow 
Installing dependencies from lock file

[ValueError]
Extra [marshmallow] is not specified.

• adding a --option marshmallow or similar flag for poetry install might be useful

Assume the following is manually added to the sample project to wrap some optional packages into an extra:

[tool.poetry.extras]
models = ["marshmallow", "pydantic"]

After this, poetry update and poetry install make no changes, they do not install the optional extra packages. This seems like useful default behavior. To get the extra packages installed, it must be explicitly requested:

$ poetry update
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

No dependencies to install or update

$ poetry install --extras models
Installing dependencies from lock file

Package operations: 3 installs, 0 updates, 0 removals

  - Installing dataclasses (0.6)
  - Installing marshmallow (3.7.1)
  - Installing pydantic (1.6.1)

$ pip freeze
certifi==2020.6.20
dataclasses==0.6
marshmallow==3.7.1
pydantic==1.6.1

This seems OK and provides a means to get all the extras installed for purposes of unit-testing etc. but it does require the extra step of adding optional packages to an explicit declaration for package extras. (I may be missing something about some alternative way to install optional packages, apologies if I missed it.)

poetry export

$ poetry export -f requirements.txt  -o tmp.txt
$ cat tmp.txt  # this is empty; no optional packages here
$ poetry export -f requirements.txt  -o tmp.txt --extras models --without-hashes
$ cat tmp.txt 
dataclasses==0.6; python_version < "3.7"
marshmallow==3.7.1
pydantic==1.6.1

So, by default poetry export does not include any extra dependencies (and no optional dependencies)

Summary of Feature Requests

• Using an --optional argument could behave like --dev to include optional dependencies
  • when --optional is specified assume it includes all the optional dependencies
• Using an --option argument could behave like an --extras option to install just one optional dependency and multiple --option arguments are allowed; when options are often grouped together then the --extras feature is more useful to declare a group of optional dependencies
• Enhance poetry show with additional flags that can filter or annotate the listings with identification of whether or not a package is installed and whether it is optional, extra, dev etc.

default export format

  -f (--format)          Format to export to. Currently, only requirements.txt is supported.

So make that the default and output to a default TmpFile that is printed in the response rather than crash like

$ poetry export

[ValueError]
Invalid export format: None

Desired behavior:

$ poetry export
exported to:  /tmp/requirements_random-string.txt

debug versions

$ poetry debug

Poetry
Version: 1.0.10
Python:  3.6.10

Virtualenv
Python:         3.6.10
Implementation: CPython
Path:           /opt/conda/envs/py3.6
Valid:          True

System
Platform: linux
OS:       posix
Python:   /opt/conda/envs/py3.6

[finswimmer]

Since Poetry 1.2 you can define more dependency groups and those can be marked as optional.