ENH: PyPy Support

[lithomas1]

Tracker for getting PyPy support

• Add PyPy wheels on MacPython/pandas-wheels and get them on PyPI(we have them already on conda-forge)
  - We can probably add this for 1.3.1.
  - Actually, it will probably be easier to add after 1.3.1, since 1.3.1 includes a fix for the PyPy build.
• Add whatsnew, putting PyPy as experimentally supported.
• Testing/CI on master(requires PyPy to release a python 3.8 version), allow failures on this build
• Xfail failing tests, and remove allowing failures on this build
• Move out of experimental support.

[simonjayhawkins]

@lithomas1 removed milestone. we can assess whether any PRs could be backported (if/when opened) otherwise the milestone is normally only used to track regressions and PRs requiring backport.

[lithomas1]

Whoops, my bad.

[mattip]

It seems the third step, enable tests here, should be the first. As I read things, the CI is currently building pandas but not testing it, and uses a PyPy from conda-forge via miniconda. So I think I should

• get a newer PyPy on conda-forge with a fix for BUG: (PyPy) sometimes find_stack_level() will stop on functools.partial #54478
• prepare a PR that expands the range of expected error messages since error messages differ
• add to that skipping tests with __sizeof__ or getsizeof which do not exist on PyPy

That should get test failures down to a point where it makes sense to

• turn on testing for PyPy in CI without coverage (since coverage slows PyPy by somewhere around 4x) and maybe without xdist (since testing seems to use too much memory)

Locally, that gets ubuntu down to 84 test failures in about 1hr22m, which is not that much longer than the other workflows under Unit Tests. Maybe more of those failures are due to unexpected error messages in failures.

xref @h-vetinari (since I think work here should precede work on the pandas-feedstock PR)

[nramprasad85]

Getting error with import pandas, I am trying to use pandas in pypy, please help me with this issue

import pandas as pd
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas_init_.py", line 22, in
from pandas.compat import is_numpy_dev as is_numpy_dev # pyright: ignore # noqa:F401
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas\compat_init.py", line 25, in
from pandas.compat.numpy import (
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas\compat\numpy_init_.py", line 4, in
from pandas.util.version import Version
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas\util_init_.py", line 2, in
from pandas.util._decorators import ( # noqa:F401
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas\util_decorators.py", line 14, in
from pandas.libs.properties import cache_readonly
File "C:\Users\Ramprasad.conda\envs\pypy_algo\lib\site-packages\pandas_libs_init.py", line 13, in
from pandas._libs.interval import Interval
TypeError: type 'cython_function_or_method' is not an acceptable base type

Here is my installation

(pypy_algo) C:\Users\Ramprasad>conda list
# packages in environment at C:\Users\Ramprasad\.conda\envs\pypy_algo:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
expat                     2.5.0                h63175ca_1    conda-forge
intel-openmp              2023.2.0         h57928b3_49503    conda-forge
libblas                   3.9.0              18_win64_mkl    conda-forge
libcblas                  3.9.0              18_win64_mkl    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              18_win64_mkl    conda-forge
libsqlite                 3.43.0               hcfcfb64_0    conda-forge
libxml2                   2.11.5               hc3477c8_1    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.25.2           py39h821adfb_0    conda-forge
openssl                   3.1.3                hcfcfb64_0    conda-forge
pandas                    2.0.3            py39haf6a60e_1    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pypy                      7.3.12                 0_pypy39    conda-forge
pypy3.9                   7.3.12               h994e1e7_5    conda-forge
python                    3.9.17                0_73_pypy    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.9               4_pypy39_pp73    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.43.0               hcfcfb64_0    conda-forge
tbb                       2021.10.0            h91493d7_1    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zlib                      1.2.13               hcfcfb64_5    conda-forge

[mattip]

This is a bug in the current PyPy feedstock. See conda-forge/pypy3.6-feedstock#109

[nramprasad85]

Its working now. Thank you @mattip

[tiwalter]

Are there any plans to publish pypy wheels by default (for every release in the future)?

[WillAyd]

It looks like conda-forge is dropping support for pypy - maybe we should also consider dropping it at this point?

https://conda-forge.org/news/2024/08/14/sunsetting-pypy/