-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory is not deallocating after using gdal translate in python #7908
Comments
GDAL doesn't generally leak memory. If you put your code in a loop, you should hopefully see that RAM usage remains stable at the max value you've noted. It is probably caches kept around. Typically the worker threads are kept in a global pool, and each thread will take at least 2 MB. |
Is there a way to free this memory ? |
not really. Some of it might be to due to RAM fragmentation: cf https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading (although I don't think this is the case here) |
I tested the translate in a loop, you are right, the memory usage is stable. |
The peak memory usage should be around the size defined by GDAL_CACHEMAX (by default 5% of usable RAM). Cf https://gdal.org/user/configoptions.html . So this is user controllable. You can generally manipulate rasters much larger than memory with GDAL |
- Changed memory allocator from default to tcmalloc - Known fix for GDAL multithreading high RAM usage, see https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading - Set max cache size to 0, see OSGeo/gdal#7908 (comment) - Added manual release of gdal Dataset to free up memory faster - Added debug logging of RAM usage to better understand RAM usage
I was testing the gdal translate functionality In python using memory_profiler.
When I look at the result of the profiler I see that the memory allocated isn't released at all.
This is example for the code I used:
And I get the following result from the profiler:
It looks like the function started with memory usage of about 80 MiB and ended up with the usage of 230.4MiB.
Should it be like this ?
I am using gdal 3.4.1.
Thanks in advance,
Tobby
The text was updated successfully, but these errors were encountered: