Document why virtual environments should (nearly) always be used, and system site-packages is considered harmful

[dimbleby]

Every time someone comes here with something going wrong after they set poetry config virtualenvs.create false the answer is "oh, you didn't want to do that". It's a footgun.

Can anything be done?

• update the docs with more cautions?
• make that option harder to use / print a big old warning when it is used?
• add a new option with meaning "I really do want to use the system environment and acknowledge that it's probably a bad idea"
• other ideas?

Personally I don't want poetry to manage my virtual environments at all, though I do want it to refuse to go ahead if I've forgotten to create the environment myself. But probably it's hard to get to there from here; and certainly not everyone has the same workflow.

[kevr]

Please do not do this. This would seriously break Docker workflows, and really, that config is up to the user; I don't see any problem with that, other than the fact that you run into users who choose to do the incorrect things on their systems.

[dimbleby]

The suggestion is not that it be made impossible to use the system environment, people who insist on doing this can continue to do so. Rather, that it more clearly and firmly be discouraged.

The problem is that many of the people who avoid using a virtual environment don't really understand what they're doing, then things go wrong, then they come here... it's a big waste of everyone's time. It would be better for those people to be guided towards a pattern that's more likely to work reliably in the first place.

(#6397 was the immediate prompt for opening this, but I'm confident there are lots of other examples)

[neersighted]

Please do not do this. This would seriously break Docker workflows, and really, that config is up to the user; I don't see any problem with that, other than the fact that you run into users who choose to do the incorrect things on their systems.

I think you are among the users who might benefit from the documentation ask here -- there is no reason to avoid virtual environments in a container, and it usually causes more harm than good to do so.

Check out #6397 (comment) for a recipe for 'seamless' use of a virtual environment in a container -- you get all the ergonomic benefits of not having to use a virtual environment path prefix, but you properly isolate your components so that you are not mixing dependency trees with unknown results.

Regarding your stance of I don't see any problem with that, other than the fact that you run into users who choose to do the incorrect things on their systems, this has historically been the project's stance. However virtualenvs.create is a footgun we hand users (often users who have come from Node or Ruby or another ecosystem without the concept of virtualenvs, and who do not understand why they are important in Python), and they do not understand the risks or consequences of opting out of an 'unnecessary' feature.

The support burden of users who had a project that happened to work with virtualenvs.create false on 1.1, and broke on 1.2 has been very large. This is likely due to 1.2 growing the dep tree and becoming moderately more choosy about versions (plus being unable to upgrade the virtualenv module while running due to a race condition that should never happen in normal operation) -- I think there is a lot of benefit to be had from pushing people away from the footgun as much as possible.

[neersighted]

I wonder if splitting the option into virtualenvs.manage (so Poetry will fail if no env is active, but not create/directly manage the env) and installer.system-site-packages (or some similar split) would make more sense in the long run.

Honestly, I'd even add a warning if installer.system-site-packages is set with a docs link, since it's rarely something you actually want if you are fully informed as to how Python import paths and environments work.

[sullivancolin]

The documentation on the official installer seems to indicate that it will be installed in a separate environment and therefore it would be safe to not use a virtualenv inside a Docker container:

We provide a custom installer that will install Poetry in a new virtual environment to isolate it from the rest of your system. This ensures that dependencies will not be accidentally upgraded or uninstalled, and allows Poetry to manage its own environment.

I can install poetry inside docker with these lines:

RUN curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0
ENV PATH "/home/vscode/.poetry/bin:$PATH"
RUN /home/vscode/.local/bin/poetry config virtualenvs.create false

When I then run pip list from system level python I get:

Package    Version
---------- -------
pip        22.2.2
setuptools 58.1.0
wheel      0.37.1

This seems like it has been effectively isolated from the system level python.

However, once I run poetry install when virtualenvs.create false been set, the install fails.

Is the documentation wrong that the official installer is not actually isolated from system level python? If not, what is the correct interpretation of what it's saying and why did installation succeed with poetry<1.2? If the documentation is wrong, then it does seem like stronger guardrails would be helpful.

[neersighted]

I wrote up a whole response but it looks like Github ate it -- sorry @sullivancolin, this one is likely less in depth, but hopefully it addresses your questions.

The official installer, or any officially supported installation method will create a separate virtual environment to install Poetry into, separate from the system (or container, in this case -- we still call it system) level environment.

You can correctly see that no Poetry-related packages are installed into your system environment. However, virtualenvs.create false also forces your project to install into the system environment, which can be a bit fraught at best. Without specifics, I won't have specific insight of course, but in short 90% of users do not need virtualenvs.create false -- there's not a reason to not use a virtual environment in a container, and it protects you against a vast list of ways Python packaging can be a headache and go awry.

[neersighted]

I wonder if splitting the option into virtualenvs.manage (so Poetry will fail if no env is active, but not create/directly manage the env) and installer.system-site-packages (or some similar split) would make more sense in the long run.

Honestly, I'd even add a warning if installer.system-site-packages is set with a docs link, since it's rarely something you actually want if you are fully informed as to how Python import paths and environments work.

This is tracked in #1945

[ewjoachim]

I think it would be useful if there was a way to tell poetry to never create a venv, but never use the system python either: to refuse to run if a venv is not set.

[neersighted]

This is a lot better with PEP 668 (https://peps.python.org/pep-0668/); it prevents the 'footgun' configurations where installing Poetry can conflict with e.g. DNF or APT. While it's still possible to have some footgun-like experiences with library/python container images, those are mostly from the much more obvious pip install poetry; pip install something-else and not the more subtle pip install poetry; apt-get install <tool written in Python>.

In other words, that is de-facto happening and users will benefit as they move to newer distros/Python versions; though we should probably verify that our installer also respects the PEP 668 markers.