Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leaks when importing problems #93

Open
anugrahjo opened this issue Dec 8, 2024 · 5 comments
Open

Memory leaks when importing problems #93

anugrahjo opened this issue Dec 8, 2024 · 5 comments
Assignees
@jfowkes
Labels
memory leak A memory leak occurs upstream This is an upstream issue

Comments

@anugrahjo
Copy link

It seems that the allocated memory (or maybe a portion of it) is never released after a problem is imported.

If I run the following code, the memory usage monotonically increases as the memory allocated for each problem imported is not deallocated. The program consumes several gigabytes of RAM (~8 GB if I skip ~30 problems that take longer times to get imported) by the time it reaches the end of the loop.

import pycutest

prob_list = pycutest.find_problems()
for prob_name in prob_list:
    prob = pycutest.import_problem(prob_name)

I use PyCUTEst 1.7.1 on macOS 14.4 with Python 3.9.13.

I saw similar issues reported before. I don't know if it's possible to be fixed since threads on the other issues mentioned this comes from upstream CUTEst or from memory leaks in Python/NumPy.
@anugrahjo anugrahjo added the bug Something isn't working label Dec 8, 2024
@jfowkes jfowkes added memory leak A memory leak occurs and removed bug Something isn't working labels Dec 9, 2024
@jfowkes jfowkes self-assigned this Dec 9, 2024
@jfowkes
Copy link
Owner

jfowkes commented Dec 9, 2024 •
edited
Loading

Thank you for reporting, we've so far been unable to ascertain exactly what is causing this memory leak and whether it is inherent to NumPy, the upstream CUTEst C interface or in fact a mistake in how PyCUTEst uses the Python C API.

I will try re-compiling PyCUTEst with the AdressSanitizer as well as running it through Valgrind and see what we can find.

@jfowkes
Copy link
Owner

jfowkes commented Dec 9, 2024

@anugrahjo I created a simple memory leak test script as follows:

import pycutest

prob = pycutest.import_problem('ALLINITC')

prob = None

Then I compiled the ALLINITC PyCUTEst C extension with -fsanitize=address and ran the script.

There are indeed lots of memory leaks, but unfortunately they are all upstream in the Python C API and NumPy:

$ LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/9/libasan.so python cutest-leak-test.py

=================================================================
==3483645==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1040488 byte(s) in 313 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5f210f in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5f210f in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5f210f in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5f210f in PyObject_Malloc ../Objects/obmalloc.c:685

Direct leak of 243077 byte(s) in 135 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5b8d8c in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5b8d8c in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5b8d8c in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #4 0x5b8d8c in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #5 0x5b8d8c in PyObject_Malloc ../Objects/obmalloc.c:685
    #6 0x5b8d8c in PyUnicode_New ../Objects/unicodeobject.c:1447
    #7 0x5b8d8c in _PyUnicode_FromUCS1 ../Objects/unicodeobject.c:2363

Direct leak of 28649 byte(s) in 19 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5c0294 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5c0294 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5c0294 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5c0294 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5c0294 in PyUnicode_New ../Objects/unicodeobject.c:1447
    #6 0x5c0294 in unicode_decode_utf8 ../Objects/unicodeobject.c:5009
    #7 0x5c0294 in PyUnicode_DecodeUTF8Stateful ../Objects/unicodeobject.c:5139
    #8 0x5c0294 in PyUnicode_FromString ../Objects/unicodeobject.c:2281

Direct leak of 26208 byte(s) in 28 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5e1c08 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5e1c08 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5e1c08 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5e1c08 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5e1c08 in _PyObject_GC_Alloc ../Modules/gcmodule.c:2237
    #6 0x5e1c08 in _PyObject_GC_Malloc ../Modules/gcmodule.c:2264
    #7 0x5e1c08 in PyType_GenericAlloc ../Objects/typeobject.c:1047

Direct leak of 23976 byte(s) in 29 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5df14f in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5df14f in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5df14f in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5df14f in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5df14f in clone_combined_dict ../Objects/dictobject.c:673
    #6 0x5df14f in PyDict_Copy ../Objects/dictobject.c:2726
    #7 0x5df14f in type_new ../Objects/typeobject.c:2447

Direct leak of 11784 byte(s) in 20 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x565975 in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x565975 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x565975 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #4 0x565975 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #5 0x565975 in PyObject_Malloc ../Objects/obmalloc.c:685
    #6 0x565975 in _PyObject_GC_Alloc ../Modules/gcmodule.c:2237

Direct leak of 8474 byte(s) in 12 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x6301f7 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x6301f7 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x6301f7 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x6301f7 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x6301f7 in _PyBytes_FromSize ../Objects/bytesobject.c:81
    #6 0x6301f7 in PyBytes_FromStringAndSize ../Objects/bytesobject.c:112

Direct leak of 3397 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x5a9925 in PyMem_RawRealloc ../Objects/obmalloc.c:590
    #2 0x5a9925 in _PyObject_Realloc ../Objects/obmalloc.c:2011
    #3 0x5a9925 in _PyObject_Realloc ../Objects/obmalloc.c:1999
    #4 0x5a9925 in PyObject_Realloc ../Objects/obmalloc.c:703
    #5 0x5a9925 in resize_compact ../Objects/unicodeobject.c:1080
    #6 0x5a9925 in _PyUnicodeWriter_Finish ../Objects/unicodeobject.c:14123

Direct leak of 3232 byte(s) in 4 object(s) allocated from:
    #0 0x7f8087c77c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x564fdc in _PyObject_GC_Resize ../Modules/gcmodule.c:2309

Direct leak of 2703 byte(s) in 2 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5b9a9b in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5b9a9b in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5b9a9b in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #4 0x5b9a9b in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #5 0x5b9a9b in PyObject_Malloc ../Objects/obmalloc.c:685
    #6 0x5b9a9b in PyUnicode_New ../Objects/unicodeobject.c:1447
    #7 0x5b9a9b in unicode_decode_utf8 ../Objects/unicodeobject.c:5009

Direct leak of 2696 byte(s) in 5 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5f13a0 in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5f13a0 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5f13a0 in _PyObject_Malloc.part.0.lto_priv.0 ../Objects/obmalloc.c:1645

Direct leak of 1677 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5bfa69 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5bfa69 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5bfa69 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5bfa69 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5bfa69 in PyUnicode_New ../Objects/unicodeobject.c:1447

Direct leak of 1582 byte(s) in 32 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5f267f in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5f267f in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5f267f in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5f267f in PyMem_Malloc ../Objects/obmalloc.c:605

Direct leak of 436 byte(s) in 41 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f8083a807a9 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x4297a9)

Direct leak of 96 byte(s) in 3 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5531ba in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5531ba in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5531ba in PyThread_allocate_lock ../Python/thread_pthread.h:392
    #4 0x5531ba in newlockobject ../Modules/_threadmodule.c:577
    #5 0x5531ba in thread_PyThread_allocate_lock ../Modules/_threadmodule.c:1170

Direct leak of 72 byte(s) in 12 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f8083901d93 in default_malloc (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x2aad93)

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x56cf60 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x56cf60 in PyThread_allocate_lock ../Python/thread_pthread.h:392

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7f8083a80ca8 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x429ca8)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7f8083a80c90 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x429c90)

Direct leak of 23 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f8083a807e8 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x4297e8)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f8083a8083d in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x42983d)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7f8083a80cc3 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x429cc3)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f8083a8084e in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/home/jfowkes/.local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so+0x42984e)

Indirect leak of 217544 byte(s) in 228 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5e1c08 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5e1c08 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5e1c08 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5e1c08 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5e1c08 in _PyObject_GC_Alloc ../Modules/gcmodule.c:2237
    #6 0x5e1c08 in _PyObject_GC_Malloc ../Modules/gcmodule.c:2264
    #7 0x5e1c08 in PyType_GenericAlloc ../Objects/typeobject.c:1047

Indirect leak of 25219 byte(s) in 18 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5df8b2 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5df8b2 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5df8b2 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5df8b2 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5df8b2 in type_new ../Objects/typeobject.c:2696

Indirect leak of 24744 byte(s) in 18 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5b8d8c in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5b8d8c in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5b8d8c in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #4 0x5b8d8c in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #5 0x5b8d8c in PyObject_Malloc ../Objects/obmalloc.c:685
    #6 0x5b8d8c in PyUnicode_New ../Objects/unicodeobject.c:1447
    #7 0x5b8d8c in _PyUnicode_FromUCS1 ../Objects/unicodeobject.c:2363

Indirect leak of 13271 byte(s) in 10 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5c0294 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5c0294 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5c0294 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5c0294 in PyObject_Malloc ../Objects/obmalloc.c:685
    #5 0x5c0294 in PyUnicode_New ../Objects/unicodeobject.c:1447
    #6 0x5c0294 in unicode_decode_utf8 ../Objects/unicodeobject.c:5009
    #7 0x5c0294 in PyUnicode_DecodeUTF8Stateful ../Objects/unicodeobject.c:5139
    #8 0x5c0294 in PyUnicode_FromString ../Objects/unicodeobject.c:2281

Indirect leak of 3376 byte(s) in 4 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5f210f in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #2 0x5f210f in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #3 0x5f210f in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #4 0x5f210f in PyObject_Malloc ../Objects/obmalloc.c:685

Indirect leak of 1512 byte(s) in 2 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x565975 in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x565975 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x565975 in _PyObject_Malloc ../Objects/obmalloc.c:1645
    #4 0x565975 in _PyObject_Malloc ../Objects/obmalloc.c:1638
    #5 0x565975 in PyObject_Malloc ../Objects/obmalloc.c:685
    #6 0x565975 in _PyObject_GC_Alloc ../Modules/gcmodule.c:2237

Indirect leak of 536 byte(s) in 1 object(s) allocated from:
    #0 0x7f8087c77808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5f13a0 in _PyMem_RawMalloc ../Objects/obmalloc.c:99
    #2 0x5f13a0 in PyMem_RawMalloc ../Objects/obmalloc.c:572
    #3 0x5f13a0 in _PyObject_Malloc.part.0.lto_priv.0 ../Objects/obmalloc.c:1645

SUMMARY: AddressSanitizer: 1684888 byte(s) leaked in 945 allocation(s).

@jfowkes jfowkes added the upstream This is an upstream issue label Dec 9, 2024
@anugrahjo
Copy link
Author

@jfowkes Thanks for the update.

I don't have many ideas here on how to pin down the exact source of this, as I'm not very familiar with the design of the interface to CUTEst. However, it looks like the leaks from NumPy are insignificant and should not be a cause for concern.

@jfowkes
Copy link
Owner

jfowkes commented Dec 12, 2024

@anugrahjo the leaks are coming from deep within Python's C API rather than our interface, e.g. _PyUnicode_FromUCS1 in Objects/unicodeobject.c:2363 is not a function we ever call explicitly. This seems like a related upstream issue:
python/cpython#113055
A quick search suggests there are lots of reports upstream of memory leaks in CPython.

@anugrahjo
Copy link
Author

@jfowkes That makes sense. After quickly going through the issue you copied above, it looks like it will not be fixed for Python versions <=12.0. Hopefully, it will be resolved in one of the patch releases for py13.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jfowkes jfowkes

Labels
memory leak A memory leak occurs upstream This is an upstream issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jfowkes @anugrahjo