-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PyMem_Raw{New,Resize}
convenience macros
#50
Comments
What's the reason to add these?
That's convenient, but on error it will forget the pointer's originaf value -- as |
When you do the multiplication, you can have an overflow. So you cannot just replace it by For downstream users, that would be helpful (otherwise the overflow check in
For the resize, I can live without it I think. |
Right, there's some value in that, though I'm not convinced there's enough value. Are there any examples of projects that would want this? The check isn't something that needs a macro, so if we want this, we should first add a function with a calloc-like signature, and then add a macro to do the macro-only part. (Or not -- users can easily type Maybe we instead want a (But I still think we don't need this. We're not building a general-purpose C library here.) |
Well, my first idea was CPython itself as a project. I don't think I'll be able to convince you more on this matter :( But I don't have more arguments except the consistency with the existing Should I tag the other members of the C API WG for voting? |
Do you have some examples? |
In static void*
BZ2_Malloc(void* ctx, int items, int size)
{
if (items < 0 || size < 0)
return NULL;
if (size != 0 && (size_t)items > (size_t)PY_SSIZE_T_MAX / (size_t)size)
return NULL;
/* PyMem_Malloc() cannot be used: compress() and decompress()
release the GIL */
return PyMem_RawMalloc((size_t)items * (size_t)size);
} In if (extra_group_size > 0) {
extra_groups = PyMem_RawMalloc(extra_group_size * sizeof(gid_t));
if (extra_groups == NULL) {
PyErr_SetString(PyExc_MemoryError,
"failed to allocate memory for group list");
goto cleanup;
}
} In /* Note: size will always be longer than the resulting Unicode
character count */
if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) - 1 < size) {
return -1;
}
unicode = PyMem_RawMalloc((size + 1) * sizeof(wchar_t));
if (!unicode) {
return -1;
} In if (raw_malloc) {
bytes = PyMem_RawMalloc((len + 1) * max_char_size);
} In if (argsize > PY_SSIZE_T_MAX / sizeof(wchar_t) - 1) {
return -1;
}
res = (wchar_t *)PyMem_RawMalloc((argsize + 1) * sizeof(wchar_t));
if (!res) {
return -1;
} And in some other places (perhaps 4-5 more). Note that there are many places where |
Note that for the Unicode-related calls, we should perhaps leave them as is since |
IMO, if we want to make arrays for us easier, the more ideal solution would be something like an internal array API, not more allocator macros. For users, I could see some usage, but in my experience, users choose either the system |
I added PyMem_RawMalloc() to be able to trace these memory allocation made by Python using the |
I'm +0 on adding PyMem_RawNew() and PyMem_RawResize(). I can see the benefits of "built-in" integer overflow checks, but I'm also used to test manually for integer overflow. I don't have a strong opinion, since it's less common to use PyMem_RawMalloc() than using PyMem_Malloc() |
We have convenience macros for
PyMem_New
andPyMem_Resize
to allocate memory for n objects of the given type without checking ifn * sizeof(T)
would overflow but we don't have any for the raw allocators. I suggest adding those two macros.@encukou expressed some reservations on the existing issue (python/cpython#127415) concerning their usefulness:
Concerning this point, there are benefits of using
PyMem_RawNew
over the existingPyMem_RawCalloc
. First,PyMem_RawNew
usesmalloc()
instead ofcalloc()
which downstream users might find convenient for their behaviour (calloc() initializes the memory being allocated while malloc() doesn't) and, possibly, for their performances (calloc() might be slightly slower). Second, using the correct semantics may be useful for downstream users as well.Maybe so, but it's a documented way to resize memory and can be used directly as
if (PyMem_Resize(...) == NULL) { ... }
which could be convenient. Again, this may not necessarily be useful for CPython itself but we do provide convenience macros for the PyMem_* APIs, so I don't know why we wouldn't do the same for the PyMem_Raw* APIs.The text was updated successfully, but these errors were encountered: