Queue watcher: memory error during tiff->nii conversion

[mhturner]

Most recently on @AlexYkHao data from 20220417. I've seen it before and seems to happen only when server + queue watcher are going at the same time, and also when a big anatomical scan is being converted.

From
dataflow_log_20220417-212808.txt

Converting tiffs to nii in directory: h:\alex\220417__queue__/fly04/anat/TSeries-12172018-1322-003
BidirectionalZ is True
isVolumeSeries is True
num_channels: 2
num_timepoints: 100
num_z: 241
num_y: 512
num_x: 1024
Traceback (most recent call last):
File "C:/Users/User/projects/brukerbridge/scripts/main.py", line 98, in
main(sys.argv[1:])
File "C:/Users/User/projects/brukerbridge/scripts/main.py", line 70, in main
bridge.convert_tiff_collections_to_nii(dir_to_process)
File "c:\users\user\projects\brukerbridge\brukerbridge\tiff_to_nii.py", line 155, in convert_tiff_collections_to_nii
convert_tiff_collections_to_nii(new_path)
File "c:\users\user\projects\brukerbridge\brukerbridge\tiff_to_nii.py", line 155, in convert_tiff_collections_to_nii
convert_tiff_collections_to_nii(new_path)
File "c:\users\user\projects\brukerbridge\brukerbridge\tiff_to_nii.py", line 155, in convert_tiff_collections_to_nii
convert_tiff_collections_to_nii(new_path)
File "c:\users\user\projects\brukerbridge\brukerbridge\tiff_to_nii.py", line 174, in convert_tiff_collections_to_nii
tiff_to_nii(new_path)
File "c:\users\user\projects\brukerbridge\brukerbridge\tiff_to_nii.py", line 63, in tiff_to_nii
image_array = np.zeros((num_timepoints, num_z, num_y, num_x), dtype=np.uint16)
MemoryError

[bellabrez]

ok yes I do believe this scan size is already very close to mazing out memory. The transfer server should take really low memory, for example the chunksize of data it holds in memory is CHUNKSIZE = 1_000_000 aka 1 MB, so this doesn't immediately make sense to me as the culprit. My guess would be other background processes (or especially chrome if its open), especially if the computer wasn't restarted in a while. Thoughts?

[mhturner]

OK that's good that server and queue aren't interfering with one-another. Beyond periodically restarting the machine I think it would be worthwhile to remove some of the crap running in the background that we don't need - the biggest hog was little chrome processes (even though chrome wasn't open, because chrome). First things first I think we should get chrome off that thing and just use firefox.

[mhturner]

Is it common for users to have images that are this big? This is like 25GB. Is it too annoying or clunky to pre-compute how much memory the array will take up beforehand, and if it's over some threshold maybe skip that image, or split it up into several sub-stacks. To avoid crashing the pipeline.

[bellabrez]

The size Alex is using here is exactly the same size I use for my anat scans. Ashley has files that are even bigger, so she uses the "split" flag in her user config file so that main.py runs convert_tiff_collections_to_nii_split instead of convert_tiff_collections_to_nii. This does what you suggest. The only annoying part is she needs to re-merge them later in on sherlock.

I think checking the expected mem size then based on a threshold call the split version if it is above the threshold makes sense. Or we can just make users set that flag. I would want to set the threshold above what the above scan is (25GB) since I actually personally haven't had that crash on me and would prefer not the extra complication of splitting and merging.

An alternative is we could save as .h5 instead of .nii which accepts saving in one or several slices at a time into the file on disk so this would work seamlessly for any size file.

I suppose it comes down to your question of how often is this actually an issue. idk if anyone other than Ashley uses really big files, and if they do maybe they can just set the flag. But I'm open to changes!