-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
C API: Consider adding a public PyList_Extend() function #111138
Comments
#define PyList_Extend(list, arg) PyList_SetSlice((list), PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, (arg)) I don't mind adding it as a macro. Together with #define PyList_Clear(list) PyList_SetSlice((list), 0, PY_SSIZE_T_MAX, NULL) |
This private function is no longer exported in Python 3.13. It is possible that a PyList_Extend() function-like macro may be added before Python 3.13 final, but using PyList_SetSlice() directly will still work. python/cpython#108451 python/cpython#111138
Just noting that the proposed |
This private function is no longer exported in Python 3.13. It is possible that a PyList_Extend() function-like macro may be added before Python 3.13 final, but using PyList_SetSlice() directly will still work. python/cpython#108451 python/cpython#111138
I dislike macros: they cannot be used in programming languages other than C, and in projects which only load libpython symbols (ex: vim text editor). See for example PEP 670 – Convert macros to functions in the Python C API. |
For a method used in Python, it makes sense. For a C API, it looks inefficient to return None on success. |
I agree, and I think returning It’s just worth pointing out that extension authors porting to Python 3.13 will need to be aware that they cannot simply search-and-replace |
Test the public PyList C API.
Test the public PyList C API.
Test the public PyList C API.
Add test_capi.test_list tests.
First step: I wrote PR #111582 to test the existing public PyList C API. |
Add test_capi.test_list tests.
Add test_capi.test_list tests.
* Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
* Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
* Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
* Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
I close the issue. Commit babb787 adds PyList_Extend() and PyList_Clear(). I wrote python/pythoncapi-compat#80 to add these functions to pythoncapi-compat. @serhiy-storchaka's macros can also be used on Python 3.12 and older: #define PyList_Extend(list, arg) PyList_SetSlice((list), PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, (arg))
#define PyList_Clear(list) PyList_SetSlice((list), 0, PY_SSIZE_T_MAX, NULL)
I didn't add this function. |
…thon#111862) * Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
Replace `_PyList_Extend` with `PyList_Extend` from `pythoncapi_compat.h`. python/cpython#111138 https://docs.python.org/dev/c-api/list.html#c.PyList_Extend Fixes ```cpp /home/runner/work/mypy/mypy/mypyc/lib-rt/list_ops.c: In function ‘CPyList_Extend’: (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/list_ops.c:259:12: error: implicit declaration of function ‘_PyList_Extend’; did you mean ‘CPyList_Extend’? [-Werror=implicit-function-declaration] (diff) 259 | return _PyList_Extend((PyListObject *)o1, o2); (diff) | ^~~~~~~~~~~~~~ (diff) | CPyList_Extend (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/list_ops.c:259:12: error: returning ‘int’ from a function with return type ‘PyObject *’ {aka ‘struct _object *’} makes pointer from integer without a cast [-Werror=int-conversion] (diff) 259 | return _PyList_Extend((PyListObject *)o1, o2); (diff) | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c: In function ‘CPyDict_Keys’: (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c:233:21: error: initialization of ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion] (diff) 233 | PyObject *res = _PyList_Extend((PyListObject *)list, view); (diff) | ^~~~~~~~~~~~~~ (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c: In function ‘CPyDict_Values’: (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c:253:21: error: initialization of ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion] (diff) 253 | PyObject *res = _PyList_Extend((PyListObject *)list, view); (diff) | ^~~~~~~~~~~~~~ (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c: In function ‘CPyDict_Items’: (diff) /home/runner/work/mypy/mypy/mypyc/lib-rt/dict_ops.c:273:21: error: initialization of ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Werror=int-conversion] (diff) 273 | PyObject *res = _PyList_Extend((PyListObject *)list, view); (diff) | ^~~~~~~~~~~~~~ (diff) ```
…thon#111862) * Split list_extend() into two sub-functions: list_extend_fast() and list_extend_iter(). * list_inplace_concat() no longer has to call Py_DECREF() on the list_extend() result, since list_extend() now returns an int.
Feature or enhancement
The private
_PyList_Extend()
function has been removed in Python 3.13: see PR #108451.@scoder asked what is the replacement for this removed function.
The obvious replacement is
PyObject_CallMethod(list, "extend", "O", arg)
: call thelist.extend()
method. But it's slower,PyObject_CallMethod()
has to get the method and decode the "O" format string.Another replacement is
PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg)
which is less straightforward, but it's efficient.I propose adding a public PyList_Extend() function. The list type is commonly used in C extensions, it's a convenient API to create a collection when the length is not known is advance.
I don't think that performance is really the problem here since
PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg)
is available. The problem is more to more the function easier to discover and make the API easier to use. Also, it should help people to migrate away from the removed function.There is already a public PyDict_Update() API for the dict type, it is part of the limited C API.
If we add a function for the list type, should we also add PySet_Update() "for completeness"? The private
_PySet_Update()
was removed in Python 3.13.cc @serhiy-storchaka @encukou
A code search on
_PyList_Extend
in PyPI top 5,000 projects (2023-07-04) found 4 projects using this function:Logs:
Cython uses the function to implement its own __Pyx_PyList_Extend() API:
Linked PRs
The text was updated successfully, but these errors were encountered: