Port to Python 3.11: use Py_SET_SIZE()

[vstinner]

On Python 3.11, "Py_SIZE(result) = cSize;" fails with a compiler
error. Since Python 3.9, Py_SET_SIZE() must be used instead:

• https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes
• https://docs.python.org/dev/c-api/structures.html#c.Py_SIZE

Add a copy of the pythoncapi_compat.h header file to get
Py_SET_SIZE() on Python 3.8 and older:
https://github.com/pythoncapi/pythoncapi_compat

[vstinner]

The build error:

building 'zstd' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -O0 -fPIC -I/home/vstinner/dev/python-zstd/env/include -I/home/vstinner/python/main/Include -I/home/vstinner/python/main -c src/python-zstd.c -o build/temp.linux-x86_64-3.11-pydebug/src/python-zstd.o -O2 -DVERSION="1.5.0.2" -DZSTD_MULTITHREAD=1 -Izstd/lib -Izstd/lib/common -Izstd/lib/compress -Izstd/lib/decompress -DZSTD_TRACE=0
src/python-zstd.c: In function ‘py_zstd_compress_mt’:
src/python-zstd.c:119:25: error: lvalue required as left operand of assignment
  119 |         Py_SIZE(result) = cSize;
      |                         ^
error: command '/usr/bin/gcc' failed with exit code 1

I'm the author of the Py_SIZE() change in Python 3.11 :-) It has been approved by the Steering Council: python/steering-council#79

I'm also working on a PEP which changes other macros and explains the rationale for the Py_SIZE() change, see: https://www.python.org/dev/peps/pep-0674/

Moreover, I'm the author of the pythoncapi_compat project and its pythoncapi_compat.h header file: https://github.com/pythoncapi/pythoncapi_compat I'm trying to propose it as a standard way to support multiple Python versions with a single C code base. The header provides new C API functions on old Python versions.

[vstinner]

Test with the development branch of Python 3.11:

$ ~/python/main/python -m venv env # Python 3.11
$ env/bin/python setup.py test
(...)
INFO:ZSTD:Python version: 3.11.0a2+ (heads/main:cb2b3c8d35, Dec  2 2021, 11:42:55) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
test_compression_default_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_default (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_zero (tests.test_compress.TestZstdCompress) ... ok
test_compression_level1 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level20 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level6 (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_many (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_one (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level_notdefault (tests.test_compress.TestZstdCompress) ... ok
test_compression_random (tests.test_compress.TestZstdCompress) ... ok
test_compression_wrong_level (tests.test_compress.TestZstdCompress) ... ok
test_library_version (tests.test_version.TestZstdVersion) ... ok
test_library_version_number (tests.test_version.TestZstdVersion) ... ok
test_library_version_number_min (tests.test_version.TestZstdVersion) ... ok
test_module_version (tests.test_version.TestZstdVersion) ... ok

----------------------------------------------------------------------
Ran 16 tests in 0.021s

OK
INFO:ZSTD:Python version: 3.11.0a2+ (heads/main:cb2b3c8d35, Dec  2 2021, 11:42:55) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
test_compression_default_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_default (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_zero (tests.test_compress.TestZstdCompress) ... ok
test_compression_level1 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level20 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level6 (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_many (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_one (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level_notdefault (tests.test_compress.TestZstdCompress) ... ok
test_compression_random (tests.test_compress.TestZstdCompress) ... ok
test_compression_wrong_level (tests.test_compress.TestZstdCompress) ... ok
test_library_version (tests.test_version.TestZstdVersion) ... ok
test_library_version_number (tests.test_version.TestZstdVersion) ... ok
test_library_version_number_min (tests.test_version.TestZstdVersion) ... ok
test_module_version (tests.test_version.TestZstdVersion) ... ok

----------------------------------------------------------------------
Ran 16 tests in 0.009s

OK

[sergey-dryabzhinsky]

Did you check it with python versions 2.7, 3.4-3.10 too?

[vstinner]

Did you check it with python versions 2.7, 3.4-3.10 too?

I tested manually the following versions (thanks Fedora for providing a wide range of Python versions!):

$ git clean -fdx; python2.7 setup.py test
$ git clean -fdx; python3.4 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.5 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.6 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.7 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.8 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.9 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.10 -m venv env; env/bin/python setup.py test
$ git clean -fdx; ~/python/main/python -m venv env; env/bin/python setup.py test # dev version of Python 3.11

=> All builds and tests passed successfully!

pythoncapi_compat.h is compatible with Python 2.7: it has been tested on Windows with Visual Studio and Python 2.7. Mercurial uses it and Mercurial is still supporting Python 2.7 on Windows ;-)

By the way, the setup.py test command emits a warning:

WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.

Using tox would avoid me to have to create the virtual environment and run Python manually for each Python version :-D

[sergey-dryabzhinsky]

Good. Thanks.