Error: Unable to allocate 932. GiB for an array with shape (353762, 353762)

[bmill3r]

Hello,

First, thanks for this great tool!

I am trying to apply it to a somewhat large single cell resolution vizgen dataset of human lung cancer (Vizgen_Human_LungCancer_Patient1) from the MERSCOPE FFPE Human Immuno-oncology data release. The dataset contains 353762 cells. When I try to run SpatialDE.run()

I receive the following error:
MemoryError: Unable to allocate 932. GiB for an array with shape (353762, 353762) and data type float64.

I think the function is generating a spatial graph of (353762, 353762) and requires 932. GiB of memory, which is way more than what I have available, even on an HPC node. Note that if I subset the data to about 25%, I get a similar same error:
MemoryError: Unable to allocate 64.8 GiB for an array with shape (93270, 93270) and data type float64

Have you come across this error before, and if so, do you have any suggestions to overcome it? Is there some way to read/write and store the data more efficiently so that I can apply SpatialDE to single cell resolution datasets? Squidpy seems to be able to handle datasets of this size, so maybe there is something they do that SpatialDE can also incorporate?

Thanks!
Brendan

[SmallHorseBrother]

This method is not suitable for situations with large datasets.