Proposal: support multiple Python versions (venv) for a same project

[reorx]

Background

Currently pipenv only works for one single Python version within one project.

For example, when creating virtualenv using command:

pipenv --python 3.6

Pipfile will contain:

[requires]

python_version = "3.6"

After pipenv install, Pipfile.lock will contain:

    "_meta": {
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "16.7.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "darwin"
        },
        "requires": {
            "python_version": "3.6"
        },
        ...,

which restricts the Python version to be 3.6.3, then, every time this project is cloned and run pipenv install, the virtualenv will be automatically created using Python 3.6.3. If the user wants to switch Python to another version, he/she would run pipenv --python 2.7 install, and the following output will occur:

Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /usr/local/bin/python2.7 to create virtualenv…
...
Warning: Your Pipfile requires python_version 3.6, but you are using 2.7.13
  $ pipenv check will surely fail.
Warning: Your Pipfile requires python_version 3.6, but you are using 2.7.13 
  $ pipenv check will surely fail.

Note that Pipfile.lock will not be changed through the process above.

In another situation, if [requires] is removed from Pipfile, the result of pipenv --python 2.7 install will be a little bit different: Pipfile.lock will be changed to 2.7.13

This is neither explicit, nor friendly to the user, because:

• When [requires] exists, the result of install cmd is not recorded, user doesn't know which file is used to specify the dependencies (Pipfile or Pipfile.lock?), and what version they are installed as.
• When [requires] not exists, Pipfile.lock is changed every time --python <version> is different, makes a lot of hassle for tracking this file in git.

In summary, Pipenv has these behaviors:

• Pipenv determines virtualenv (dir) name by full path of the Pipfile, as long as the project path not change, the virtualenv (dir) name will always be the same.
• Pipfile.lock can only contain one Python version (environment markers) at a time.
• Pipfile.lock's Python version may or may not change when running install cmd according to Pipfile's [requires] definition

Proposal

The best way to solve problems above is to make Pipenv aware to Python version, possible changes are listed as below:

1. Pipenv determines virtualenv dir name with python version. If currently a virtualenv name is calculated as project-Tl8cuoWa, using new algorithm it should be project-Tl8cuoWa-3.6.3.
2. Lock file should be environment bound, files should be put under Pipfile.lock/<version>. For example, if run --python 3.6 install, new lock file will be created as Pipfile.lock/3.6.3, and install without specified --python <version> will be rejected as parameters missing.
3. Pipfile requires should be a list ([[requires]]), allowing to set multiple Python versions (this could be ignored because requires is optional)
4. --python <version> should only let Pipenv select which virtualenv should be used as target, not rm and re-created entirely.
5. Add ensure command, used for ensure target virtualenv's dependencies are installed as its lock file defined.

So the workflow will be like:

1. When initialize a new project, use pipenv --python <version> to create Pipfile, with one item in [[requires]]. This will not rm & re-create existing virtualenv.
2. When adding dependencies, use pipenv --python <version> install <dep> to install. This will not rm & re-create existing virtualenv.
3. When cloning and preparing new environment, use pipenv --python <version> ensure to install dependencies and ensure consistency with lock file. This will not rm & re-create existing virtualenv.
4. When running comand, use pipenv --python <version> run <cmd>. This will not rm & re-create existing virtualenv.

References

The following issues point out related problems this fork wants to solve:

• multiple parallel environments pypa/pipenv#1071
• Support for multiple virtual environments (tox) pypa/pipenv#368
• Support multiple python_version pypa/pipenv#1050

[tonybaloney]

@reorx https://github.com/tonybaloney/tox-pipenv

[gsemet]

I would be happy to have something like:

$ pipenv install --python '>=3.5'

that set a requires section such as

[requires]
python_version = ">=3.5"

allowing global python to work like package markers. This would also be legal:

[requires]
python_version = ">=3.5;<3.7"

[reorx]

@gsemet the problem of this issue, or the shortcoming of pipenv, has already been resolved by poetry (the word "resolve" may not be accurate, since poetry is designed to work with multiple python major versions at the very beginning).

After a short period of trial with poetry, I can confirm this is what a dependency management tool for python should be. The design philosophy is sane, the spec is clear, it tries to do one thing, and does it pretty well so far.

Actually, after I've created this issue with my thoughts been expressed, I found it would be very difficult to solve them thoroughly in pipenv. Binding virtual environment with dependency management is not a good decision (just take a look at import this). There are fundamental problems in pipenv's design concept, it's not the tool aimed purely at managing dependencies, it's just a collection of tools and workflows that help you working with python projects, in the way that the author think is nice (which is not to everyone and not even to a slightly formal use case). So I decided to give up enhancing pipenv, but think again on what the community actually needs, and wait for the new comer.

Now poetry has finally came, it is indeed what I need, maybe not the best, but the purpose is right enough to go on, and the quality is good enough to start using it today.