-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rapidtide failing due to memory issue #10
Comments
Setting
|
@bbfrederick have you profiled rapidtide based on the input file size in the past? E.g., rapidtide requires X times the size of the input file's array? |
I had an old rapidtide run on my laptop actually. The |
That's probably about right. I don't know how strongly the scalar depends on the various options, but that's not a bad place to start.
~Are you running this in a container, on bare metal on your local machine, or on a cluster?~ (nvm - I see you are running it in Docker) --nprocs sets the number of cpus to "all the cpus on the machine" if you give it an argument of -1, it will use all the cpus on the node, so on clusters, and Docker containers, where you aren't able to use all of them, you need to set it explicitly. Because of the way python forks processes, that increases your memory usage some (although the majority of allocations are done in shared memory, so the memory load is not linear with the number of cpus).
BTW, Github is seeming kind of broken to me - very long load times, can’t save comments on the issues page.
… On Sep 4, 2024, at 10:50 AM, Taylor Salo ***@***.***> wrote:
I had an old rapidtide run on my laptop actually. The memusage's Self Max RSS column peaks at 8.069 GB, and my input file's data array is ~1.5 GB, so a scalar of ~5.5x might be roughly what rapidtide needs. @bbfrederick <https://github.com/bbfrederick> does that sound right?
—
Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH5DU5XULZ4TUM6RSX2GDLZU4M45AVCNFSM6AAAAABNS3WY4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRZGI3TSMZWHA>.
You are receiving this because you were mentioned.
|
I'm testing it out on my local machine (an old MacBook Pro) with Docker. I have Docker limited to 6 GB of memory because anything more would probably make my machine unusable while it's running. I have nprocs set to 1. Are there any other tricks I should employ to save memory? |
You can try --spcalculation - that does all internal math in single precision to save RAM. Also, call rapidtide with --memprofile - this will help figure out where the problem is. |
One thing that has always annoyed me about RAM use - macOS and linux python implementations seem to handle internal memory allocation very differently - if you run rapidtide on macOS directly RAM use goes up AND DOWN over time. But in any linux environment, including Docker containers running on macOS, RAM use only goes up, and doesn't ever seem to reuse deallocated RAM. By the end of a run on a linux system, rapidtide can have something like 50GB of VMEM when processing an HCP-YA resting state dataset. BTW, rapidtide shouldn't need all its RAM to be resident at once. Does maximizing Docker's swap space help? |
I tried using
I know it's a problem that memprofiler isn't installed in the Docker image, but the error seems unrelated to that. I'll try increasing the swap space next time. |
That last failure seems to be because memprofiler wasn't installed but it just barreled along and tried to use it anyway. I'll check that and fix it. If |
That's a great idea! I'll try that out. EDIT: I'll need an interface for retroglm anyway, since we'll eventually want to loop over output spaces and denoise in each. |
Is retroglm what we will use to apply the regressor to other spaces (e.g., CIFTIs)? |
I hadn't thought about it, but yes, that could certainly work. You'd just need to resample the delay map and the mask to the target space. |
The more I think about it, the more I'm convinced that we should run rapidtide in boldref space and then warp the outputs to other spaces, both because it's more memory-efficient and because it's easier to warp from boldref to other output spaces using fMRIPrep transforms. I'm trying that out locally and it seems to be working. |
Is there a gray/white/CSF segmentation in boldref space by default? If not, we'd need to generate one. |
I can just warp the T1w/T2w-space one into boldref space. |
What happened?
It's probably something I can fix with my fmripost-rapidtide call or my Docker settings
What command did you use?
What version of fMRIPost-Rapidtide are you running?
main
How are you running fMRIPost-Rapidtide?
Docker
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
Anatomical derivatives
Please copy and paste any relevant log output.
Additional information / screenshots
No response
The text was updated successfully, but these errors were encountered: