Support multiple python_version

[antoine-de]

Hi,

I would like to support multiple python_version (more exactly, I want to support python 3.5 and 3.6, but there is a problem with python 3.5.2 on a sub dependency, so I would like the support >3.5.2) and I can't find the right way to define this.

In the documentation it says Pipenv automatically honors both the python_full_version and python_version PEP 508 specifiers. but I don't seems to be able to express it:

in my pipenv I put:

[requires]
python_full_version = ">3.5.2"

> pipenv install
Warning: Your Pipfile requires python_version >3.5.2, but you are using 3.5.3 (/home/a/.local/share/v/a/bin/python)

I see that request does not define the required python version, it seems to be done in travis. Is it the right way to do this ?

[techalchemy]

This is a good suggestion, we should be able to implement this fairly easily

[gsemet]

+1

[kennethreitz]

no

[gsemet]

it would help a lot

[isobit]

@kennethreitz I (and hopefully others here) would appreciate further explanation than just "no". This seems like such an obvious feature, especially since the documentation explicitly says pipenv supports the PEP 508 specifiers.

[kennethreitz]

you're asking a lot out of me, and i've already put a tremendous amount of time into this, and it's a holiday right now. patience is a virtue.

[gsemet]

that's because we love pipenv :) But i think this will help the adoption of pipfile/pipenv for some libraries or application, especially if one want to support Python 3.5+

[kennethreitz]

it will be considered

…

On Wed, Nov 22, 2017 at 2:17 PM, Gaetan Semet ***@***.***> wrote: that's because we love pipenv :) But i think this will help the adoption of pipfile/pipenv for some libraries or application, especially if one want to support Python 3.5+ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/kennethreitz/pipenv/issues/1050#issuecomment-346448207>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHUVc2MSERTYmXgwbnh8wD4qAInLE__ks5s5HM8gaJpZM4QYD9C> .

[isobit]

I'm not asking for anything other than an explanation for why the issue was closed. I appreciate the huge effort you've put in and I totally understand that things like this aren't going to be implemented overnight. However, when the issue is just closed like this, the signal I get as a user is that this will never get fixed, so I was curious as to why.

[djetelina]

Right now, this means that running multiple python versions in CI fails. travis-ci example:

python:
- "3.6"
- "3.7-dev"
install:
- pip install pipenv
- pipenv install --dev

then running pipenv check results in:

$ pipenv check .
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Checking PEP 508 requirements…
Specifier python_version does not match 3.6 (3.7).
Failed!

Which is a shame.

[uranusjr]

@iScrE4m Just remove the python_version requirement from Pipfile if your intend is to support multiple Python versions.

[gsemet]

@uranusjr how do we tell our package works for python 3.5 and 3.6 but not 3.4 or 2.7 ?

[foutoucour]

@gsemet Do you think you can mention it in your setup.py to inform your users?
If you need a strict restriction, you might want to have a check in your code that double check the current version of python.

What do you think?

[gsemet]

i don't see the difference with the python_version markers for dependencies. the whole role of this is to ease automatisation, CI/CD, auto test and so; so that we minimize the among of "what to know" in the readme. Of course we "put a few line" somewhere to inform the user, but i wish we had something like

[required]
python_version = ">=3.5"

so that it is crystal clear for pipenv and everything that will read the Pipfile that, no, my package won't work in 3.4 or 2.7 and fail with a nice error.

Same for example if my package works in 3.5 and 3.6 but not on the upcoming 3.7 (python_version>=3.5;<3.7), we can describe by range so that pipenv install --dev automatically discard 3.7, for instance, and install 3.5 or 3.6, depending on the availability on the system

[uranusjr]

This seems like another chance to mention Donald’s wonderful writeup on this:
https://caremad.io/posts/2013/07/setup-vs-requirement/

Donald was describing package dependencies, but the same principle applies for the dependency of Python itself. Pipfile (and requiremets.txt, the format it aims to replace) is for applications (webapps, bundled client applications, etc.), where you would only support one pinned version at a time. Information about pip-installable libraries should go into setup.py (or setup.cfg). In your case, your package seems to be intended to be installed by a non-controlled audience, using a variety of Python versions. This falls into the library category, and related metadata should therefore go into setup.py, not Pipfile.

Also, specifying required.python_version in Pipfile also does not really guarentee anything. With the proposed solution, your pipenv check calls on Travis CI would work, but there is still nothing preventing someone running a non-supported Python version from installing your package, because Pip does not read Pipfile. It reads setup.py and setup.cfg. To achieve your ultimate goal (preventing installation on incompatible versions), you need to put the requirement in setup.py or setup.cfg.

Now, like most packages out there contain both a setup.py and a requirements.txt, there is absolutely no problem have a Pipfile in your package repository, and use it via Pipenv. Pipenv itself does that as well. In that scenario, however, the Pipfile would simply manage the development environment. It does not have anything to say about how your package would be installed, and should not be used for that. Specify those in setup.py or setup.cfg instead.

[isobit]

Does pipenv consider the Python version from setup.py or setup.cfg when using pyenv to manage the dev environment? My use case was wanting to specify the version for pipenv to install using pyenv for dev without also breaking usage when not using pyenv (and using the system Python instead).

[gsemet]

I do not completely agree with this article. I actually use PBR to abstract this setup.py-ish complexity and works great for both libraries and applications (it is widely used on many OpenStack projects). Hardcoding stuff in setup.py should be considered as bad practices. It is hard to parse, external services (such as readthedocs,...) cannot easily access to this information,... So if setup.py can be as simple as possible for maintainability sake, i'll be happy.

requirements.txt is a great improvement over hardcoded setup.py (even find_packages is hard). It allowed external tools to work with the package, and so much confort in edition.

My point is: Libraries ship with requirements.txt describing compatibility versions by range, applications ship with requirements.txt freezing all versions. I used a lot pip-tools to maintain requirements.txt.in, and Pipfile/pipenv makes life much easier now: libraries does not track Pipfile.lock, applications does. Work done. But for the moment all application does not support Pipfile* (first of all: PBR itself), so I keep requirements.txt synchronized (with version not frozen for libraries, and with version frozen for applications).
So for the dependencies part, the job is pretty much done without having to do kung-foo in setup.py.

Pipfile/pipenv is the future of python, and that is greatly expected!

Just for the interpreter itself, sadly, there are limitation: this file can say "it works for Python 3.5", but there is no way in Pipfile to tell "this pipfile works for python 3.5 or 3.6", for example.

TL;DR: Just a "or". We only would like to have a simple "or" syntax for python interpreter
It is a small improvement that would help a lot.

[techalchemy]

In short this is handled by setuptools and it seems odd to call the packaging standard a bad practice but you should raise that issue there. We don’t parse setup.py or setup.cfg but instead simply allow pip’s internals to do so.

Setup files are for installing a package. As an end user if I want to install a thing you made, your setup file tells my package manager (setuptools) where to put scripts and what dependencies I need. Requirements files don’t do anything besides list dependencies, so relying on one for distributing your software doesn’t make a ton of sense. Whatever abstraction you choose for handling these things, this will still be the case.

[jmfederico]

I'm just starting to use Pipenv and run into this "issue".

I am developing Cookiecutter template which makes use of F-Strings, hence it requires python > 3.6, but there is no way for me to specify that on Pipfile.

I do not want to force a version, just need to specify a minimum requirement, and can't do it in a setup.py file, because this is not meant to be installed, pipenv is to be run within the folder of the project.

When I "pipenv install" I need to remember to use "pipenv --python 3.6 install", and I don't see that as intuitive for anyone else that uses the project.

I think it would be incredibly helpful to be able to declare python compatible versions, and I would greatly appreciate if it gets implemented.

😄

[techalchemy]

@jmfederico you can absolutely specify this in the Pipfile, that's precisely the origin of this issue. Using the existing syntax you can precisely specify that your pipenv environment is going to be installed using python 3.6 by putting the following section in your pipfile:

[requires]
python_version = "3.6"

This ensures that when you run pipenv install it will use python 3.6. The issue you are commenting on is a request to support specifying multiple python versions in that section

[jmfederico]

But what I want is it to be minimum 3.6, but if the user does not have 3.6 but 3.7, it should use 3.7.

Right now it gets fixed to any 3.6.x ve version and nothing else. No way to say >3.6.

[techalchemy]

@jmfederico I don't know much about cookiecutter but I would imagine you can put in your code, which will be executed somewhere, either some kind of fallback or throw an exception if the user attempts to use it in a non-python3.6 environment. If you are distributing your environment itself then currently the expectation is you are using a specific version of python with it, but pipenv wasn't designed as a package distribution tool which is why this feature wasn't built in

Not saying it won't ever be included, but dependency resolution for multiple minor python versions at once in the same lockfile might not even be possible

[jmfederico]

I could throw an exception if python <3.6 (in my case) is used, but it just makes it cumbersome for the user. If the user has a compatible python version installed, it would be better for him/her if pipenv used it.

I do think that this is something that will happen with more than a fair share of projects, and handling it directly within pipenv would avoid repetition. It could be the same as when one specifies a given version, and pipenv complains that it can’t find the version that is being requested, just that it would say that it can’t find any compatible version.

It would move the exception handling up the tool chain.

For now local exception it is, but, as I said, would be awesome (at least for me) if it was handled by pipenv.

[techalchemy]

@jmfederico I can't speak for other maintainers but the more I hear arguments in favor of a universal tool for handling all packaging related operations in python, the more I am nudged in that direction. These kinds of things feel out of scope while the resolver still has issues but I can see a good argument for supporting these kinds of things in the future

[gsemet]

I use marker for that (ex: install typing for python <3.5), and it is already supported by pipenv.
What you mean is that we would need one Pipfile.lock per environment, since in the current state a lock file is for only one environment. But once the lock file is generated, the environment is quite reproductible (not fully, see bellow).

Lock file are for application and we expect application environment to be fully reproductible on a given environment. So having an interpreter version and all the pypi packages frozen in the lock file makes sense.
For libraries, I think we can all agree there is now way to achieve build reproductibility. I wonder if using Pipfile.lock for library build only (and frozen version should not reflect on install_requires, somehow) could be good, I have sometime variation between the resolution on my machine and on the Travis builds that is not easy to understand.

I would also love to be able to generate several lock files (ex: Pipfile.lock.2.7, Pipfile.lock.2,5 and Pipfile.lock.3.5), to accommodate to the deployment environment (not sure if that is enough, some packages such as numpy would be machine architecture dependent).

I start freezing the version of pip and pipenv in my sidecar makefile in my applications (i heavily use docker) because recent updates of pip and pipenv broke the build, and since we use lockfile to prevent unwanted update on pypi to broke a previously valid build, it is not acceptable to have pipenv broke a fuly working environment on a update.

And that’s ok for pipenv and pip to evolve and sometime break compatibility. And bugs happen, that’s life.
What is currently missing to pip/pipenv ecosystem is a resistance feature, since it is primarily install on the system and not the virtualenv.

Maybe we might want to have several version of pipenv side by side in the user/system environment so that the Pipfile can lock the pipenv/pip version has well. It might be complex to do but in the current state the builds on python using pip and pipenv aren’t reproductible enough even with the lock file. That maybe subject to another thread.

[Ariki]

I still don't understand the maintainers' point of view. @techalchemy points out that pipenv is not a package distribution tool and user should use specific Python version. But pipenv does not give us real control over Python version, we can only chose one of Pythons provided by the OS distribution (or installed/compiled manually). It's not like rustup utility which allows you to download and install specific version of Rust to get a reproducible environment. So we need at least some flexibility from the tool that positions itself as an easy method to set up a working environment. By using markers in requirements.txt, I can, for example, have a single dependency list in my project that allows me to get a working environment under Python 2.7, 3.5+, and PyPy 6.0, and even switch between these environments. It's weird that I can't do this with a Pipfile. And all the flexibility that we have in specifying package versions is of no use if we are allowed to use just one version of interpreter. I like the idea to use a single Pipfile for a flexible dependency list and several lock files for reproducible environments.

[ankitbhatia8993]

Warning: Your Pipfile requires python_version >=3.6, but you are using 3.6.4 (/Users/abc/.local/share/v/t/bin/python).
$ pipenv check will surely fail.
What is the reason of this warning?

[lietu]

@ankitbhatia8993 it seems the reason is because the developers are not interested in other people's problems, and in their experience Python applications never get distributed outside of your own controlled environment.

Must be too much to ask for them to explain more than "no" ( #1050 (comment) ) during a period of a year to something that seems quite obvious to most people, but in another issue someone seems to spout nonsense about how it's best practice to support only one minor version of Python in an "application context" (while most people are perfectly fine saying their code runs on Python >=3.5 and then know to code against Python 3.5).

#2683 (comment)

Right now your only solution is to remove the related section from your Pipfile.

[kennethreitz]

@ankitbhatia8993 that seems like a bug, possibly.

[kennethreitz]

@lietu please drop the attitude. Pipenv is for application deployments, not libraries.

[lietu]

@kennethreitz Who said anything about libraries? Since when is it not allowed to distribute applications with support for multiple Python versions?

Is support for >=3.5 going to kill you or make Pipenv somehow worse? I doubt it. There is an attitude problem involved with this, and it's yours.

Write an APPLICATION using e.g. various tools such as Kivy, Django, Qt, and you will find that your application tends to support a wide range of Python versions, and you don't always need to ship your own.

Most obviously this problem applies to web applications. You build your application happily on Python 3.6.3, and lo and behold Python 3.6.4 comes out on your server's distro's repos and your application breaks because you have to specify the exact version of Python you support.

You as the developer probably didn't care at all. Mostly thought "oh nice" if you read it has some security fixes. And instead your builds are now failing.

Right now your available options to avoid this issue are:

1. Bump into this issue, decide Pipenv is shit and use something else
2. Bump into this issue, realize you can remove that section (which is btw added there by default), and be left without the protection it can also provide you with - e.g. in case a dev tries to run things without realizing their system is only running Python 2.7
3. Write some sort of extra things around Pipenv to make Pipenv behave the way you want it to work for your environment, and every other environment you ever plan to distribute your application into

APPLICATIONS get distributed as well, and are affected by other things in the world. They don't just stay in your perfectly controlled secret vault deep in a cave somewhere.

Just give one good reason to not support it, "my applications never leave my controlled environment" is not one.

[lietu]

Some additional thoughts:

The 3.6.3 -> 3.6.4 upgrade was hastily written and probably wrong, either way applies to 3.5 -> 3.6 upgrade and it doesn't need to.

When I write e.g. open source applications I expect that someone is going to run it on Windows, someone on macOS, someone on Ubuntu, and someone on CentOS, then where easily possible I try to accommodate.

This typically means that I would probably write a version requirement spec similar to >=2.7,<3.0|>=3.3,<4.0, and then e.g. use Travis-CI and/or tox to check that it actually works with all of the versions I advertise. Why I can't simply do this, is incomprehensible to me.

I also don't understand the reasoning to limit your view of appropriate uses of Pipenv to just applications, and not libraries. Why build such artificial limitations on what your tool can support? You could just let the users decide what they find the tool useful for, and if there's e.g. issues preventing use with libraries put them in the issue tracker.

I can fully understand the attitude "This doesn't affect me, so I'm not going to fix it.", but I cannot comprehend the attitude "This doesn't affect me, so I'm going to close the issue and no-one will get a fix ever."

[techalchemy]

Unfortunately the decision wasn’t made based on whether it impacts one single maintainer. The design decisions about Pipfile specifications are based on the intent of the project. You say that when writing applications you like to specify a wide range of versions and platforms. That tells me you didn’t understand the explanation I offered earlier— covering a range of python versions and platforms is precisely the behavior of a library, and identifies the boundary we keep pushing here.

This constraint is applied for applications which are things you deploy. The point of an application is that you deploy using the precise environment you developed it with. Once you start distributing your software you are now discussing a library which is the thing we are telling you to use setup.py or setup.cfg for. The simple answer to your question is this: lockfiles are for idempotent environments. Idempotent environments are simply not possible when you start crossing python versions and platforms. Therefore it is not supported by pipenv to generate lockfiles this way.

[techalchemy]

I should add this: just because we didn’t articulate something in this specific issue doesn’t mean we didn’t articulate it or we made some arbitrary decision because we feel like it. Please don’t come to the issue tracker with a negative attitude and attempt to point fingers and place blame and accuse us of not caring and being selfish. We spend a lot of time supporting this project, mostly on things none of us personally use, and none of us is paid for that time. We are very invested in making good decisions about pipenv and we think carefully about which things to support and why, and often there are things we attempted to support and found we couldn’t due to technical limitations.

Basically, just because you can’t understand something doesn’t mean there wasn’t a good reason, so to come to our issue tracker and accuse us of doing it out of apathy or malice is a clear indicator that you aren’t here for productive discussion. If you do want to talk productively, feel free to open a new issue describing the feature or bug you’d like to talk about. I get your frustration that things are not how you would like for them to be. Consider that you are not the only person who has ever looked at this and wondered if it were possible, though. If you think you can make an idempotent, fully reproducible, platform and python version independent lockfile, we would absolutely review that PR

[lietu]

The point of an application is that you deploy using the precise environment you developed it with.
Once you start distributing your software you are now discussing a library

First time I've heard that to be the definition of an application or a library. I'd go as far as to say it simply sounds like you're wrong about that.

I think Ansible, Salt Stack, and e.g. GateOne are applications, that need to work with a number of different Python versions due to being deployed to various different environments.

Some applications do bundle their own full environments with them, others don't. Both are perfectly valid options.

Therefore it is not supported by pipenv to generate lockfiles this way.

Who said anything about lockfiles?

It's simple a matter of Pipfile supporting the PEP 508 syntax for defining the required Python version. Generally most libraries already support a wide range of Python versions so you can just have the exact same dependencies regardless of if you are running Python 3.5 or 3.6.

Please don’t come to the issue tracker with a negative attitude and attempt to point fingers and place blame and accuse us of not caring and being selfish.

It was high time someone mentioned that saying "no" and closing the issue is not an appropriate thing to do.

Basically, just because you can’t understand something doesn’t mean there wasn’t a good reason, so to come to our issue tracker and accuse us of doing it out of apathy or malice is a clear indicator that you aren’t here for productive discussion

I'm clearly trying to argue the case for this feature request being discussed here. It seems you are simply trying to claim that you are right and everyone else is wrong, case closed, no need for further elaboration. Which one of us is against productive discussion, really?

The amount of times this seems to have popped up as a thing people think should obviously be supported tells something.

If you were to add the support for PEP 508 syntax, what would change for you?

You could still use e.g. >=3.6,<3.7 for locking it to 3.6.x and get what you wanted. On top of that, everyone else would be happy too.

[techalchemy]

This is both a technical constraint and a design decision. You can claim that 'supporting pep 508 syntax' would be simple, but that is not right. I have already explained why. No amount of claiming that people support it will solve this issue which I have described several times. This is not a matter of convincing anyone this is a good idea.

I think Ansible, Salt Stack, and e.g. GateOne are applications, that need to work with a number of different Python versions due to being deployed to various different environments.

Do you pip install it? It's packaged as a library. Let me point you at this article about the distinction which is referenced throughout the issue tracker and which describes why libraries are specifically the things you put on PyPI.

It seems you are simply trying to claim that you are right and everyone else is wrong, case closed, no need for further elaboration. Which one of us is against productive discussion, really?

I am attempting to have a productive discussion, but you are simply restating that it is a 'simple matter' of supporting pep 508 syntax, which means you didn't actually read my response explaining why it is not at all simple.

Who said anything about lockfiles?

Considering that the entire function of pipenv is to create idempotent environments, the state of which is represented in a Pipfile.lock, the impact of any design decision on lockfiles seems pretty critical. If that is not clear then you need to spend some time learning about pipenv before demanding features and attempting to explain how simple their implementation would be. Again, if you have a working implementation, I'd be glad to review your PR. Thanks for your feedback.

[lietu]

You can claim that 'supporting pep 508 syntax' would be simple

What I meant is that the task simply means supporting PEP 508 instead of whatever else you seemed to try to bring in for seemingly no reason, and did not imply it's necessarily easy to do.

You also don't close issues because they're hard to solve, you close them because they're resolved or invalid. If it's a hard problem, you leave the issue open, and if you're feeling kind you also leave it with an explanation on why it's hard, maybe a link to the relevant source with a message saying "if anyone feels daring enough to give it a try" or similar.

Also having an issue open does not mean someone is expecting you to work on it immediately, just that it's been acknowledged as a problem that someone can work on in the future. If you're scared it does, GitHub has tags to say e.g. Low Priority or Problematic etc.

Do you pip install it?

No.

E.g.:

curl -L https://bootstrap.saltstack.com | sudo bash -s -- git develop

Basically what that does on CentOS is described at https://github.com/saltstack/salt-bootstrap/blob/develop/bootstrap-salt.sh#L3608-L3983 .. lots of things, but none of them is pip install salt.

There are lots of OSS things out there that run on a wide variety of Python versions, and you install with git clone, or by downloading a .tar.gz and then run, without any pip install <application> involved. Hey here's even something I wrote back in the days that is an application, had Travis tests for multiple Python versions, and is not published on PyPI: https://github.com/lietu/twitch-bot

I do comprehend the difference between things you publish to PyPI and things you install as applications. For one you typically want setup.py, and for the other you can do whatever you want. Considering what kind of a mess setup.py tends to be, it wouldn't be a terrible idea to see a tool such as Pipenv to provide an alternative to that in the future.

the impact of any design decision on lockfiles seems pretty critical

I see one mention of this version in the Pipfile.lock: ._meta.requires.python_version. Hey I could be wrong, but with my relatively layman's point of view (in terms of this codebase) it doesn't seem like it means a massive change for the lockfile.

Again, if you have a working implementation, I'd be glad to review your PR.

Sure doesn't look like it, and will not look like it to anyone, since the issue is closed. THIS is the whole point.

The issue was closed with the message "no". That indicates to everyone who comes to this page (quite a few people from the looks of it) that the team simply doesn't care.

Anyway, at this point I've done all I can to hammer in the point. If you still don't get it/care, there is no point to me wasting my time on this, as I'm also not paid to do this and there's also a limited amount of things I'm willing to do for free.

[jmfederico]

I have to say I am really confused.

Please correct me if I am wrong:

• The whole point of a lock file is to provide a reproducible environment,
• but Pipfile is not yet a lock file,
• Pipfile.lock IS the lock file,
• Pipfile allows me to set a base point of requirements from which I can then create a Pipfile.lock which I can then commit so that I know what environment my app is running against, and doing it so reliably, which I can then reproduce.
• How is allowing a relaxed version requirement for python in Pipfile affecting Pipfile.lock?
• Seems like it is exactly the same as allowing a relaxed requirement for any other package.

[erinxocon]

I've been taking a break from pipenv, but this seems like a super non-contriversal issue to jump back into... I will try and add a few things.

If you want your package to support multiple python versions, put it in your setup.py and let it be installed by the user. That's were it belongs; in setup.py. Pipfile/Pipfile.lock/pipenv are not meant to replace a setup.py or a setup.cfg. It is a replacement for requirements files only. To my knowledge requirements.txt doesn't even allow you to specify a minimum python version for the project only per a requirement according to pep508. If you are distributing your project to other people for them to use as a library, write a setup.py. If you distributing your project to other people for them to run as an application, use Pipfile and pipenv. Bonus points if you have pyenv installed as pipenv will try and download the version of python specified in the Pipfile so it works exactly as it did on the creators machine.

[erinxocon]

There are some great packages that help aid in packaging like https://github.com/takluyver/flit. They use pyproject.toml which is intended to replace the setup.py. Pipenv doesn't try and do this, it's just virtualenv and dependency management tool, could be called a packaging tool for applications, but not libraries.

[techalchemy]

Also we close issues because this is the issue tracker where we decide what is going to be incorporated into code or not. This is not planned as a feature, not a bug, etc. it is a conscious choice to not pursue it, not simply ‘because it is hard’, but because we chose not to pursue it.

Beyond that we can actually close issues for a variety of reasons besides merging code that addresses them, but you can start your own project and handle issues on that project any way you choose.

To put a finer point on it: we have thought about the implications of this decision. We have no plans to change it. Thanks for understanding.