Train multiple models in a single GPU on parallel

[Programmer-RD-AI]

Discussed in #10159

^{Originally posted by grudloff October 27, 2021}
Is there a recommended way of training multiple models in parallel in a single GPU? I tried using joblib's Parallel & delayed but I got a CUDA OOM with two instances even though a single model uses barely a fourth of the total memory. And is a speedup compared to sequential calling expected?

[Programmer-RD-AI]

Solution https://pytorch-lightning.readthedocs.io/en/stable/advanced/multi_gpu.html

[Programmer-RD-AI]

Solution #2807

[Programmer-RD-AI]

Solution https://wandb.ai/wandb/wandb-lightning/reports/Multi-GPU-Training-Using-PyTorch-Lightning--VmlldzozMTk3NTk

[awaelchli]

Hey @Programmer-RD-AI
Please don't duplicate posts from the discussion forum here into GitHub issues.

GitHub issues are for:

• Bug reports
• Feature requests
• Anything related to the development of Lightning

GitHub issues are not:

• For pure question answering / implementation help
• Broad discussions not directly related to the PL development

Thanks for your understanding.