0.21.0 wheels broken for older (<1.13) numpy version

[jorisvandenbossche]

xref #16715 (comment)

I can reproduce what was reported in the other issue: in a clean env, first installing python -m pip install numpy==1.11.2 and then python -m pip install pandas (0.21.0) results in a broken install, while in principle our wheels should be build against the oldest supported numpy version and this should thus work.

From a quick glance it however seems that the python 2.7 wheels are still build with numpy 1.7: https://github.com/MacPython/pandas-wheels/blob/master/.travis.yml

cc @TomAugspurger @jreback

[jorisvandenbossche]

My assumption is this: for all python 3 versions we specify a build numpy version of at least 1.19.3 (NP_BUILD_DEP=numpy==1.9.3), but for the python 2.7 build we do not specify this, so in practice the numpy version is there 1.7.1.
Since this version is no longer supported by pandas, during the wheel building a newer version of numpy will be installed, and the python 2.7 wheels are then actually build against latest numpy 1.13, hence those errors on older numpy versions.

[jreback]

note this just broke in the last day or 2

[jorisvandenbossche]

MacPython/pandas-wheels#19

[jreback]

I pushed some updates, lets see.

[jorisvandenbossche]

note this just broke in the last day or 2

Why would you say that?
There already have been reports two weeks ago in the other issues

[jreback]

no our builds just broke
the wheels themselves are a separate issue

[TomAugspurger]

It seems like uploading wheels to PyPI with the same name is a no-go. We would need to to bump the version somehow, perhaps with .post0, and then do another release with that. But, since we're close to a 0.21.1 release anyway, I'd say we put up with this for another few days.

Users who want the "fixed" wheels with NumPy < 1.13 can install directly from the rackspace CDN, when the builds for MacPython/pandas-wheels@ec160e0 finish:

pip install -f "https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com" pandas

[jorisvandenbossche]

pip install -f "https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com" pandas

This one doesn't seem to work (I also don't see any 0.21.0 packages on there, only dev builds and rc).
The link in the travis log is http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com/. However installing from there with pip also does not work because it is not a trusted source.

I did download the wheel from there manually, and verified the wheel is not working properly though (you need to add --force-reinstall --upgrade --no-deps --no-cache to the pip install to update your broken version).

no our builds just broke
the wheels themselves are a separate issue

Sorry, I don't understand. We only have one version of the wheels uploaded for 0.21.0, and they were broken from the beginning, no?

[ogrisel]

There is a HTTPS link for the wheels container as well:

https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com/

[jorisvandenbossche]

OK, so the full pip install command needed to install this version is:

python -m pip install pandas==0.21.0 --force-reinstall --upgrade --no-deps --no-cache --find-links https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com/ --no-index

(note the --no-index in the end, as otherwise it will still download the pypi one)

[jreback]

no our builds just broke
the wheels themselves are a separate issue

the daily wheel generation has been working up until 3 or 4 days ago, its fixed now with the 2.7 numpy version pin.

The only wheel that is an issue is the 2.7 one. as @TomAugspurger and I discussed in person, its not feasible to update this inplace on PyPi. the 0.21.1 release will fix this. Note we should have a small note in 0.21.1 whatsnew to indicate the problem is resolved.

[jorisvandenbossche]

the daily wheel generation has been working up until 3 or 4 days ago, its fixed now with the 2.7 numpy version pin.

OK, but this issue was about the released version, not the daily one.

I know we cannot update the wheel. Depending on how much time it will take, it's ok for me to wait until 0.21.1 (but that time might depend on the sum discussion)

[jreback]

to be honest we should just release 0.21.1 soon as-is, pending the matplotlib fix issue only (the import warning);

[jorisvandenbossche]

0.21.1 wheels are up, and should now be working.

[amir20]

@jorisvandenbossche if I understand correctly, version 0.21.1 should have fixed the bug where pandas installed before numpy would result in ImportError: numpy.core.multiarray failed to import.

However, I still get the same error, using Docker:

$ docker run -it python:3.6-alpine sh
$ apk add --no-cache g++
$ pip install pandas
  Downloading pandas-0.21.1.tar.gz (11.3MB)
...
$ python -c 'import pandas'
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 26, in <module>
    from pandas._libs import (hashtable as _hashtable,
  File "/usr/local/lib/python3.6/site-packages/pandas/_libs/__init__.py", line 4, in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
  File "pandas/_libs/tslib.pyx", line 1, in init pandas._libs.tslib
ImportError: numpy.core.multiarray failed to import

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 35, in <module>
    "the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

If I do pip install numpy first then it does work. Is that the only work around? I am new to pandas so perhaps I am doing something wrong. Any help is appreciated either way.

[jorisvandenbossche]

@amir20 You are downloading the source files as tar.gz, not wheels. This issue was about a problem with how the wheels were built, but if you are building from source yourself, you still need to ensure that numpy is installed first.

[TomAugspurger]

And you're getting source files since you're on alpine linux. The wheels on PyPI are manylinux wheels, which are not compatible with alpine: pypa/manylinux#37

…

On Sun, Dec 17, 2017 at 1:38 AM, Joris Van den Bossche < ***@***.***> wrote: @amir20 <https://github.com/amir20> You are downloading the source files as tar.gz, not wheels. This issue was about a problem with how the wheels were built, but if you are building from source yourself, you still need to ensure that numpy is installed first. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18530 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIpiGSb89QUIt58CZa53AVQMfVdlTks5tBMT-gaJpZM4QsSpH> .

[amir20]

Oh that makes a lot more sense. 👍🏻