-
Notifications
You must be signed in to change notification settings - Fork 16
cellfinder crashes as it maxes out RAM[BUG] #52
Comments
Something has gone wrong here, the cell detection part doesn't use that much RAM (I've analysed 500GB images on a laptop with 16GB RAM). Could you try downgrading
|
Hello,
|
5 days is a very long time. Is your data on a local or a network drive? There is a lot of data read/write within cellfinder, so the faster the storage, the quicker it is. If you can put your data on a local SSD, you may see speedups. How many cells do you have? If you have many tens or hundreds of thousands, cellfinder isn't very efficient currently see brainglobe/cellfinder#356). Lastly, there isn't currently a way to speed up the classification. It only uses 3 cores, but the GPU is the bottleneck at this point. @dstansby the newest version of |
I'll take a look into this and do some memroy profiling across the changes we've been making recently. |
Here's some rough memory profiles I ran using the small dataset used in the tests: Verison 0.2.8Version 0.3.0It looks like there's two things to investigate:
Note that I haven't done multiple runs here, but this gives me a starting place for further investigations. |
I think I've found the issue - in |
Describe the bug
Hello, I recently installed cellfinder as in the instructions and run it on a clerared whole-brain (2 channels, 500+GB each). It completed the registration successfully, but crashed shortly after during the cell detection. The issue appears to be the incapacity to load further arrays in memory. (manually checking resource use with task manager confirmed saturation of ram before crash)
I have tried limiting the memory with the
--max-ram
setting, but even with--max-ram=250
(half of total ram), cellfinder ends up using the whole memory anyway and eventually crashing.When I try to use cellfinder-napari on a subset of images (16GB in total), the detection works fine.
Is there a way to actually limit ram usage by the cell detection step?
To Reproduce
I have attached my conda environment .yaml (renamed as txt) cellfinder.txt. I ran:
conda activate napari-env
cellfinder -s W:\nobackup\garber\grifalbe\20220202_FULL\C1 -b W:\nobackup\garber\grifalbe\20220202_FULL\C2 -o W:\nobackup\garber\grifalbe\test_full_brain2 -v 3 1 1 --orientation ipr --debug --atlas allen_mouse_25um --max-ram 250
Expected behavior
I hoped to run cell detection on the whole brain and that max-ram would limit ram usage of all steps of the pipeline, thus preventing it from crashing.
Log file
I am attaching the log of the last failed run.
cellfinder_2022-05-11_10-39-52.log
Desktop:
Windows Version 10.0.19042
500GB RAM,
2x Intel(R) Xeon(R) CPU E5-4669 v4 @ 2.20GHz, 2195 Mhz, 12 Core(s)
(no GPUs)
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: