FIX: support python311

[tacaswell]

In 18b5dd68c6b616257ae243c0b6bb965ffc885a23 / python/cpython#31530 / https://bugs.python.org/issue46836 the _frame struct was moved to an internal header, however the public API is primarily read-only.

I am not sure that this is the correct fix, but it gets scipy to compile again!

Description

Suggested changelog entry:

FIX: include internal CPython hearders

[Skylion007]

This isn't the right fix. The proper fix is just to not access the member of PyFrameObject directly anymore, but instead use the new limited C-API getters for newer versions of Python. (I don't think we write to any of the _frame fields so the new read only API should be just fine).

There are some other breaking changes for Python 3.11 that need to be fixed as well such as in: #3694

update : fixed link

[Skylion007]

@tacaswell Can you paste the full error if this include is missing? I suspect it's just from f_back in error_string() that we need to make use the new API?

[tacaswell]

  creating build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft
  INFO: compile options: '-DPOCKETFFT_PTHREADS -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c'
  extra options: '-std=c++14 -pthread -fvisibility=hidden -msse -msse2 -msse3'
  INFO: g++: scipy/fft/_pocketfft/pypocketfft.cxx
  In file included from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:15,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                   from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
  /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/type_caster_base.h:517:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
    517 |             frame = frame->f_back;
        |                          ^~
  In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:42,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/common.h:211,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                   from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                   from scipy/fft/_pocketfft/pypocketfft.cxx:15:
  /home/tcaswell/.pybuild/bleeding/include/python3.11/pytypedefs.h:21:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
     21 | typedef struct _frame PyFrameObject;
        |                ^~~~~~
  error: Command "g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPOCKETFFT_PTHREADS -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c scipy/fft/_pocketfft/pypocketfft.cxx -o build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o -MMD -MF build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o.d -std=c++14 -pthread -fvisibility=hidden -msse -msse2 -msse3" failed with exit status 1

Re-running just the g++ gives

$ g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPOCKETFFT_PTHREADS -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include -I/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/nump
y/distutils/include -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c scipy/fft/_pocketfft/pypocketfft.cxx -o build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o -MMD -MF build/temp.linux-x86_64-3.11/scipy/fft/_pocketfft/pypocketfft.o.d -std=c++14 -pthread -fvisibility=hidden -msse -msse2 -msse3
In file included from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:15,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                 from scipy/fft/_pocketfft/pypocketfft.cxx:15:
/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/type_caster_base.h:517:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  517 |             frame = frame->f_back;
      |                          ^~
In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:42,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../detail/common.h:211,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:13,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                 from /home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                 from scipy/fft/_pocketfft/pypocketfft.cxx:15:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pytypedefs.h:21:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
   21 | typedef struct _frame PyFrameObject;
      |                ^~~~~~

so I think this is the relevant part of the log.

[Skylion007]

@tacaswell I have added the proper changes to #3694. Can you verify that this solves the issue?

[tacaswell]

Closed in favor of not using private API via #3694 .