Synching vs Features in Workspaces

[mitsuhiko]

There is an issue which asks for sync to be implied (#114). There is now also a flag called --features and --all-features on sync/lock to opt-in certain optional dependencies (#121). I think it's a good time to think about how this can work.

The lockfile long term should work feature independent. That means that the lockfile is the same no matter which features are turned on. This is also how this works in Rust and I quite like that. So let's disregard the locking mechanism for now. The question however is about how the virtualenv is managed:

1. When you sync with --features, it remembers the state of the features turned on and next time running sync runs with the same features turned on.
2. When you sync, you need to always pass --features or it will install without optional features. That's also already the case with --no-dev which is needed explicitly to be passed to turn off the dev dependencies.
3. There is a .rye.toml file or a part in pyproject.toml which influences how sync/lock operate.

I'm a bit conflicted here. My hunch is that option 3 is probably the most sensible.

[T-256]

Hello Armin, thanks you for starting on simplify managing python projects. Here is my idea:

poetry/pdm/hatch supports multiple environments. why not rye?
this will add --env to cli commands and removes --dev from add/remove commands and --no-dev from sync command.

Dev env:

adding and syncing (--env=default by default):

rye add flask
rye add black --env=dev
rye add ruff --env=dev --optional=ruff
rye sync
rye sync --env=dev --all-features

pyproject.toml

[project]
name = "test-wsl"
version = "0.1.0"
description = "Add a short description here"
dependencies = ["flask~=2.3.2"]
readme = "README.md"
requires-python = ">= 3.8"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.rye.env.dev]
dependencies = ["black~=23.3.0"]
[tool.rye.env.dev.optional-dependencies]
ruff= ["ruff~=0.0.265"]

├── README.md
├── pyproject.toml
├── requirements-dev-ruff.lock
├── requirements-dev.lock               # default value of `feature` ignored
├── requirements.lock                   # default value of `feature` and `env` ignored
└── .venvs                              # {env}{feature}{toolchain}
    └── default-main-cpython_3.11.1
         └── bin
         └── lib/site-packages
              └── flask
    └── dev-main-cpython_3.11.1
         └── bin
         └── lib/site-packages
              └── flask
              └── black
    └── dev-ruff-cpython_3.11.1
         └── bin
         └── lib/site-packages
              └── flask
              └── black
              └── ruff

[T-256]

3. There is a .rye.toml file or a part in pyproject.toml which influences how sync/lock operate.

if using this way, we also need a command:

rye set sync --all-features

[tool.rye.args]
sync = ["all-features"]

and for reset using without any options:

rye set sync

[mitsuhiko]

poetry/pdm/hatch supports multiple environments. why not rye?

It can, but I believe it's important to get the important things right first. Rust has --release for a reason as a special form of target, and I believe this to be a crucial aspect of getting stuff work well. It's always possible to go to more environments afterwards, but that's a 1% of users type of problem.

In particular creating multiple separate virtualenvs just raises the question how tools and editors should interact with it.

[T-256]

but that's a 1% of users type of problem.

The problem is already exist but users don't care about it. Check black example, after syncing, the main dependencies and black installed together, just for formatting current project.
Actually rye could add multi env at first instead of adding dev-dependences. so we don't need install black every time using sync command.
Thats true there is --no-dev option for sync but still I think multiple env usage is more than you think.

Imagine we have multiple scenarios for format, lint, doc and testing. already added them to dev-dependices and now it's an update for ruff linter. if update it in pyproject.toml and try to syncing it will re-create venv with installations of whole current project and its dependencies, testing, document, formatting dependencies. do we need to wait for all of these to finished each time?

Also multiple environments can make it easier to use matrixes for testing.

In particular creating multiple separate virtualenvs just raises the question how tools and editors should interact with it.

yes, that's the only downside of multi envs in project folder, but there should be few ways:

1. using .direnv as it supported by vscode.
2. create all venv folders in root folder instead of subfolders of .venvs
3. using global path that symlinked to all created in all projects

[T-256]

perhaps multiple env could solve #114 too by increasing auto sync perf when targeting specific env in add/remove commands:

rye add httpx  # it will autosync to `main-default-cpython_3.11` env
rye add h2 --optional=http2  # it will autosync to `http2-default-cpython_3.11` env

# using new env that is not installing `httpx` and current project:
rye add ruff --env=lint  # it will autosync to `main-lint-cpython_3.11` env

[mitsuhiko]

Black should not be installed as a dependency to begin with. From where I stand that is a global tool and should be installed with rye install as a global tool. Maybe there should even be workspace local tool virtualenvs which just symlink the tools into the local virtualenv too.

I don't disagree that multi env stuff has an appeal, but there are much more important fish to fry for a start and multiple environments make everything more complex.