Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install pandas 0.24.2 from source with Python 2.7 into an environment without NumPy #27435

Closed
didip opened this issue Jul 17, 2019 · 32 comments
Labels
Build Library building on various platforms
Milestone

Comments

@didip
Copy link

didip commented Jul 17, 2019

Code Sample, a copy-pastable example if possible

RuntimeError: Python version >= 3.5 required.

Problem description

Under this docs: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.24.0.html

0.24.x is supposed to be backward compatible to Python 2.

The 0.24.x series of releases will be the last to support Python 2. Future feature releases will support Python 3 only. See Plan for dropping Python 2.7 for more details

Expected Output

0.24.2 should still work on Python 2.

@jreback
Copy link
Contributor

jreback commented Jul 17, 2019

are you sure you actually tried to install 0.24.x? and not the 0.25rc1

@didip
Copy link
Author

didip commented Jul 17, 2019

Looks like it is trying to download numpy that is really new.

Our specified version for numpy is numpy==1.16.2

      File "/tmp/pip-install-E4jz9w/pandas/setup.py", line 746, in <module>
        **setuptools_kwargs)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
        replace_conflicting=True,
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
        return cmd.easy_install(req)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/code/.cicd/runtime/python/.python/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/code/.cicd/runtime/python/.python/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-dglvOb/numpy-1.17.0rc2/setup.py", line 31, in <module>
    
    RuntimeError: Python version >= 3.5 required.

@jreback
Copy link
Contributor

jreback commented Jul 17, 2019

right so numpy 1.17 is also python 3 only. guess would be that you have a too old pip, or are doing some odd type of installation.

@didip
Copy link
Author

didip commented Jul 17, 2019

Inside setup.py, should this block of logic check for Python version first and set max version of numpy for Python 2?

min_numpy_ver = "1.13.3"
setuptools_kwargs = {
    "install_requires": [
        "python-dateutil >= 2.6.1",
        "pytz >= 2017.2",
        "numpy >= {numpy_ver}".format(numpy_ver=min_numpy_ver),
    ],
    "setup_requires": ["numpy >= {numpy_ver}".format(numpy_ver=min_numpy_ver)],
    "zip_safe": False,
}

@TomAugspurger
Copy link
Contributor

Can you show the command you're running? You shouldn't be executing setup.py, pip would be doing it for you if needed.

@didip
Copy link
Author

didip commented Jul 17, 2019

Fairly standard procedure: pip install -r requirements.txt.

...
numpy==1.16.2
pandas==0.24.2
...

During pip install, when it's collecting pandas...

Collecting pandas==0.24.2... 

pandas ended up downloading much newer numpy: numpy-1.17.0rc2 instead of what I specified in requirements.txt: numpy==1.16.2. numpy==1.16.2 is downloaded correctly.

@TomAugspurger
Copy link
Contributor

What platform? Are you passing --pre or --no-binary anywhere? I'm confused why you aren't getting a wheel for NumPy and pandas.

@TomAugspurger
Copy link
Contributor

You might also post the full output.

@didip
Copy link
Author

didip commented Jul 17, 2019

Collecting pandas==0.24.2 (from redacted==0.0.0)
  Downloading https://pypi.redacted.com/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz (11.8MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-E4jz9w/pandas/setup.py", line 746, in <module>
        **setuptools_kwargs)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
        replace_conflicting=True,
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
        return cmd.easy_install(req)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/code/.cicd/runtime/python/.python/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/code/.cicd/runtime/python/.python/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/code/.cicd/runtime/python/.python/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-dglvOb/numpy-1.17.0rc2/setup.py", line 31, in <module>
    
    RuntimeError: Python version >= 3.5 required.

requirements.txt

blinker==1.4
Flask==0.10.1
Flask-KVSession==0.4
Flask-Statsd-Ext==0.0.3
gevent==1.1.1
greenlet==0.4.9
gunicorn==19.4.5
itsdangerous==0.24
Jinja2==2.8
MarkupSafe==0.23
mysql-connector-python-rf==2.0.4
numpy==1.16.2
pandas==0.24.2
psycopg2==2.7.1
pyjwt==1.6.4
pylru==1.0.9
python-dateutil==2.5.0
python-ldap==2.3.13
pytest==3.3.0
pytest-cov==2.5.1
pytz==2016.3
raven==6.6.0
redis==2.10.5
requests==2.10.0
simplekv==0.7.1.dev1
six==1.10.0
SQLAlchemy==1.3.3
Werkzeug==0.11.9
Unidecode==0.4.21

setup.py

from setuptools import setup, find_packages


import os

here = os.path.abspath(os.path.dirname(__file__))
README = open(os.path.join(here, 'README.md')).read()

exclude_dirs = ['venv', 'scripts']

with open('requirements.txt') as f:
    install_requires = f.read().splitlines()


setup(name='redacted',
    long_description=README,
    packages=find_packages(exclude=exclude_dirs),
    include_package_data=True,
    scripts=[],

    install_requires=install_requires,
    dependency_links=[],
    tests_require=[],
    zip_safe=False
)

And we call python setup.py test to build our app, installing dependencies, etc.

Note that I am redacting a lot of business logic related stuff, but the flow looks like that.

@TomAugspurger
Copy link
Contributor

So pip install -r requirements.txt works?

@didip
Copy link
Author

didip commented Jul 17, 2019

pip install -r requirements.txt failed because pandas tried to download numpy-1.17.0rc2

@didip
Copy link
Author

didip commented Jul 17, 2019

Should the setup.py be updated like this? Essentially check Python 2 version?

import sys

min_numpy_ver = "1.13.3"
max_numpy_ver = ""

if (sys.version_info < (3, 0)):
    max_numpy_ver = "1.16.4"

numpy_install_rule = "numpy >= {numpy_ver}".format(numpy_ver=min_numpy_ver)
if max_numpy_ver:
    numpy_install_rule += ", <= {numpy_ver}".format(numpy_ver=max_numpy_ver)

setuptools_kwargs = {
    "install_requires": [
        "python-dateutil >= 2.6.1",
        "pytz >= 2017.2",
        numpy_install_rule,
    ],
    "setup_requires": [numpy_install_rule],
    "zip_safe": False,
}

@didip
Copy link
Author

didip commented Jul 18, 2019

The above snippet works on our internal fork of pandas. All of our Py 2 apps have correct dependencies now.
What do you guys think of updating the setup.py?

@jreback
Copy link
Contributor

jreback commented Jul 18, 2019

there are no more updates planned to the 0.24.x branch

closing this issue as won't fix.

@jreback jreback closed this as completed Jul 18, 2019
@jreback jreback added the Build Library building on various platforms label Jul 18, 2019
@didip
Copy link
Author

didip commented Jul 19, 2019

Why not publish 0.24.3? There is no other changes beside setup.py. This way people who are unfortunately stuck on Python 2 won't have broken deployment.

@TomAugspurger
Copy link
Contributor

TomAugspurger commented Jul 19, 2019 via email

@maxsond
Copy link

maxsond commented Aug 1, 2019

I ran into this as well and figured out how to work around it without code modifications thanks to the comments here. I was originally building a Docker image from alpine:3.7, and in that container pip downloaded archive files for Numpy and Pandas rather than wheels. But when I switched my container image to python:2.7-stretch instead, then pip downloaded wheels for the versions it was supposed to and everything installed fine (I haven't tried python:2.7-alpine, though).

So although if it was up to me I'd probably apply the proposed setup.py change just for ease of use, the actual source of the problem is external to pandas. If I was going to chase this further I'd probably start with pip's source and see how it would handle running in the alpine:3.7 image to figure out why it would select archive files in that situation.

@TomAugspurger
Copy link
Contributor

Just to summarize the workaround: if you're seeing this issue with Python 2.7, you need to ensure that you have NumPy installed in the environment pip install pandas is run.

@jorisvandenbossche
Copy link
Member

Note that this is very easy to run into, simply doing

pip install pandas==0.24.2 --no-binary :all:

in a clean env with python 2.7 and latest pip gives the error (I am using --no-binary to mimic a platform that has no wheels).

Does anybody know what is actually the underlying reason, or if this has been reported to pip / setuptools?

@jorisvandenbossche
Copy link
Member

OK, there is pypa/pip#6667.

So if I understand correctly, this is (within the limitations of setuptools handling of setup_requires) not an issue due to pip or how people are using it (installing should work without first needing to install the dependencies), but something we can fix on our end.

I think we should consider doing a 0.24.3 release for this.

@TomAugspurger
Copy link
Contributor

What's the fix? Adding a pyproject.toml to 0.24.3?

@jorisvandenbossche
Copy link
Member

Adding a "< 1.17" to numpy in setup_requires (and probably as well to install_requires, and probably only if you're running python 2).

That's the basically the only change they did in the "pandas27" (Tuxprogrammer#1, only need to look at changes in setup.py)

@jorisvandenbossche
Copy link
Member

The proper fix is to add a pyproject.toml, but let's try to do that again on master first and not in a bug-fix release. But as long as we don't use pyproject.toml, that means we don't get the proper build isolation of pip and we need to work with legacy setuptools / setup_requires.

@TomAugspurger
Copy link
Contributor

Adding a "< 1.17" to numpy in setup_requires (and probably as well to install_requires, and probably only if you're running python 2).

It would have to be Py2 only right? Else you can't install 0.24.2 with Python 3 and NumPy 1.17.

Is anyone actually going to do the release though? I suppose we can cut corners and not build conda packages. Not sure if we can skip building docs. But we have to build both an sdist & wheels, even though the issue is just with the sdist.

@didip
Copy link
Author

didip commented Sep 11, 2019

@TomAugspurger I have a patch to setup.py above for both Py 2 and 3.

import sys

min_numpy_ver = "1.13.3"
max_numpy_ver = ""

if (sys.version_info < (3, 0)):
    max_numpy_ver = "1.17"

numpy_install_rule = "numpy >= {numpy_ver}".format(numpy_ver=min_numpy_ver)
if max_numpy_ver:
    numpy_install_rule += ", < {numpy_ver}".format(numpy_ver=max_numpy_ver)

setuptools_kwargs = {
    "install_requires": [
        "python-dateutil >= 2.6.1",
        "pytz >= 2017.2",
        numpy_install_rule,
    ],
    "setup_requires": [numpy_install_rule],
    "zip_safe": False,
}

@jorisvandenbossche
Copy link
Member

@didip would you like to do a PR for this?

@jorisvandenbossche jorisvandenbossche added this to the 0.24.3 milestone Sep 11, 2019
@TomAugspurger
Copy link
Contributor

TomAugspurger commented Sep 11, 2019 via email

@didip
Copy link
Author

didip commented Sep 11, 2019

@jorisvandenbossche yes I can send a PR.

@jorisvandenbossche
Copy link
Member

@didip that would be very welcome!

@TomAugspurger TomAugspurger changed the title Version 0.24.2 is suddenly Python 3 only. I was under the impression that 0.24.x supports both Python 2 and 3. Cannot pandas 0.24.2 from source with Python 2.7 into an environment without NumPy Oct 15, 2019
@TomAugspurger TomAugspurger changed the title Cannot pandas 0.24.2 from source with Python 2.7 into an environment without NumPy Cannot install pandas 0.24.2 from source with Python 2.7 into an environment without NumPy Oct 15, 2019
@WillAyd
Copy link
Member

WillAyd commented Oct 22, 2019

At this point we don't have any further plans for a 24.x release, so unfortunately I don't think this is something we will address (if anyone disagrees feel free to reopen)

@WillAyd WillAyd closed this as completed Oct 22, 2019
@jorisvandenbossche
Copy link
Member

For people reading this: there is an open PR fixing this, but that needs some further updates to get CI passing.
If people are still interested in seeing this fixed, you can contribute to that PR: #28511

@hudasami
Copy link

hudasami commented Apr 3, 2020

This is how it worked for me for a similar problem:
Python Version : 2.7.12
First download .whl of compatible NumPy : https://pypi.org/project/numpy/1.12.1/#files
Install using pip install "above file"
Then download .whl of compatible pandas : https://pypi.org/project/pandas/0.20.1/#files
Install using pip install "above file"

StefanRijnhart added a commit to EssentNovaTeam/pandas that referenced this issue Apr 30, 2020
StefanRijnhart added a commit to EssentNovaTeam/pandas that referenced this issue Apr 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build Library building on various platforms
Projects
None yet
Development

No branches or pull requests

7 participants