Virtual packages cannot have optional-dependencies

[arv-anshul]

Steps to Reproduce

1. Create a optional dependency in pyproject.toml.

rye add --features=docs mkdocs-material mkdocs-markdownextradata-plugin

• Now, the pyproject.toml contains...

[project]
# more fields...
dependencies = ["bs4==0.0.1", "httpx==0.26.0", "python-dotenv==1.0.0"]

[project.optional-dependencies]
docs = ["mkdocs-material>=9.5.9", "mkdocs-markdownextradata-plugin>=0.2.5"]

2. Now, lock/sync with docs dependencies (docs reference):

rye lock --features=docs
# OR
rye sync --features=docs

Expected Result

The requirements.lock file must contains the mkdocs-material and mkdocs--markdownextradata-plugin packages for installation.

Actual Result

Lockfile(s) does not contains the docs dependency packages for installation.

The actual requirements.lock file:

# generated by rye
# use `rye lock` or `rye sync` to update this lockfile
#
# last locked with the following flags:
#   pre: false
#   features: ["docs"]
#   all-features: false
#   with-sources: false

anyio==4.2.0
beautifulsoup4==4.12.3
bs4==0.0.1
certifi==2024.2.2
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
idna==3.6
python-dotenv==1.0.0
sniffio==1.3.0
soupsieve==2.5

Version Info

rye 0.22.0
commit: 0.22.0 (203a0f33f 2024-02-09)
platform: macos (aarch64)
self-python: [email protected]
symlink support: true

Stacktrace

No response

Edited: I actually written the Expected Result in Actual Result and vice-versa. 😞

[mitsuhiko]

I cannot reproduce this. Here the section from the pyproject.toml:

[project.optional-dependencies]
docs = ["mkdocs-material>=9.5.9", "mkdocs-markdownextradata-plugin>=0.2.5"]

Lock without features:

$ rye lock && cat requirements.lock
Generating production lockfile: /private/tmp/y/requirements.lock
Generating dev lockfile: /private/tmp/y/requirements-dev.lock
Done!
# generated by rye
# use `rye lock` or `rye sync` to update this lockfile
#
# last locked with the following flags:
#   pre: false
#   features: []
#   all-features: false
#   with-sources: false

-e file:.

Lock with features:

$ rye lock --features=docs && cat requirements.lock
Generating production lockfile: /private/tmp/y/requirements.lock
Generating dev lockfile: /private/tmp/y/requirements-dev.lock
Done!
# generated by rye
# use `rye lock` or `rye sync` to update this lockfile
#
# last locked with the following flags:
#   pre: false
#   features: ["docs"]
#   all-features: false
#   with-sources: false

-e file:.
babel==2.14.0
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
ghp-import==2.1.0
idna==3.6
jinja2==3.1.3
markdown==3.5.2
markupsafe==2.1.5
mergedeep==1.3.4
mkdocs==1.5.3
mkdocs-markdownextradata-plugin==0.2.5
mkdocs-material==9.5.9
mkdocs-material-extensions==1.3.1
packaging==23.2
paginate==0.5.6
pathspec==0.12.1
platformdirs==4.2.0
pygments==2.17.2
pymdown-extensions==10.7
python-dateutil==2.8.2
pyyaml==6.0.1
pyyaml-env-tag==0.1
regex==2023.12.25
requests==2.31.0
six==1.16.0
urllib3==2.2.0
watchdog==4.0.0

What also confuses me a bit is this command you posted:

rye add --features=docs mkdocs-material mkdocs-markdownextradata-plugin

That's not a valid command and if it were, it would not produce the pyproject.toml file you pasted. The correct command would be this one here:

$ rye add --optional=docs mkdocs-material mkdocs-markdownextradata-plugin
Added mkdocs-material>=9.5.9 as optional (docs) dependency
Added mkdocs-markdownextradata-plugin>=0.2.5 as optional (docs) dependency

[arv-anshul]

Yes, you are right about this command.

# wrong command
$ rye add --features=docs mkdocs-material mkdocs-markdownextradata-plugin

# right command
$ rye add --optional=docs mkdocs-material mkdocs-markdownextradata-plugin

Reproducible commands:

Note: I am using --virtual flag here.

$ rye init --virtual --no-readme
success: Initialized virtual project in /Users/iarv/Developer/learn-rye/.
  Run `rye sync` to get started

$ rye add --optional=docs mkdocs-material
Added mkdocs-material>=9.5.9 as optional (docs) dependency

$ rye lock --features=docs  && cat requirements.lock
Initializing new virtualenv in /private/Developer/learn-rye/.venv
Python version: [email protected]
Generating production lockfile: /private/Developer/learn-rye/requirements.lock
Generating dev lockfile: /private/Developer/learn-rye/requirements-dev.lock
Done!
# generated by rye
# use `rye lock` or `rye sync` to update this lockfile
#
# last locked with the following flags:
#   pre: false
#   features: ["docs"]
#   all-features: false
#   with-sources: false

The pyproject.toml file:

[project]
# more fields...
dependencies = []
requires-python = ">= 3.8"

[project.scripts]
hello = "learn_rye:hello"

[project.optional-dependencies]
docs = [
    "mkdocs-material>=9.5.9",
]

[tool.rye]
managed = true
virtual = true
dev-dependencies = []

Important

When I try to reproduce this without --virtual flag or when I remove the tool.rye.virtual key from pyproject.toml it works fine and lockfile generated as requires.

[mitsuhiko]

That is a known limitation but it's not documented. For now I will add it to the docs.

[arv-anshul]

Thanks @mitsuhiko to clarify! 👍

Ok then after mentioning this issue in the docs, close this issue.

[arv-anshul]

Ok then after mentioning this issue in the docs, close this issue.

Ohh, you have done it already in (a4b0016).

[mitsuhiko]

I will reopen it as it's still a limitation that should eventually be fixed.

[tmke8]

This seems like exactly the use case PEP 735 is intended for:

This PEP specifies a mechanism for storing package requirements in pyproject.toml files such that they are not included in any built distribution of the project. This is suitable for creating named groups of dependencies

Would look like this:

[project]
# more fields...
dependencies = []
requires-python = ">= 3.8"

[project.scripts]
hello = "learn_rye:hello"

[dependency-groups]         # <- new
docs = [
    "mkdocs-material>=9.5.9",
]

[tool.rye]
managed = true
virtual = true
dev-dependencies = []

For a virtual package, extras are just "dependency groups" anyway.