Issue with running on multiple GPUs

[mackpack]

I am having an issue with running CycleGAN on multiple GPUs. It works well when running on a single GPU (albeit very slowly, as expected) using

 python3 train.py --dataroot ./datasets/cezanne2photo --name cezanne2photo_cyclegan --model cycle_gan

Now when I try to train on multiple GPUs using

python3 train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan --gpu_ids 0,1,2,3 --batch_size 16 --norm instance

I have also tried to run it with and without the --norm instance parameter and
also tried with --batch_size 4. This always leads to the same result:

The program stops at "create web directory" (I've let it run for a couple of days at this point, without any noticeable progress). It looks like a single python3 process is putting a single thread under full load, none of the other python3 processes get any CPU time. Trying to kill that process also seems impossible - I have had to restart the machine every time. None of the GPUs are ever under load and barely any of their memory is used.

I am using Python 3.5.2, CUDA 9.2, pytorch 1.0, cuDNN 7.4.1. The system has four 1080ti GPUs and an AMD Ryzen Threadripper 1950X.

[junyanz]

Your command with multi-GPU training works for me. I am using Python 3.6.4, PyTorch 0.4.1, CUDA 9.0, cudnn 7.0.5. @taesungp

[tangtao1999]

I am having an issue with running CycleGAN on multiple GPUs. It works well when running on a single GPU (albeit very slowly, as expected) using

 python3 train.py --dataroot ./datasets/cezanne2photo --name cezanne2photo_cyclegan --model cycle_gan

Now when I try to train on multiple GPUs using

python3 train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan --gpu_ids 0,1,2,3 --batch_size 16 --norm instance

I have also tried to run it with and without the --norm instance parameter and
also tried with --batch_size 4. This always leads to the same result:

The program stops at "create web directory" (I've let it run for a couple of days at this point, without any noticeable progress). It looks like a single python3 process is putting a single thread under full load, none of the other python3 processes get any CPU time. Trying to kill that process also seems impossible - I have had to restart the machine every time. None of the GPUs are ever under load and barely any of their memory is used.

I am using Python 3.5.2, CUDA 9.2, pytorch 1.0, cuDNN 7.4.1. The system has four 1080ti GPUs and an AMD Ryzen Threadripper 1950X.

It also happen to me. How to solve this? Guys, i need your help!

[banyet1]

I am having an issue with running CycleGAN on multiple GPUs. It works well when running on a single GPU (albeit very slowly, as expected) using

 python3 train.py --dataroot ./datasets/cezanne2photo --name cezanne2photo_cyclegan --model cycle_gan

Now when I try to train on multiple GPUs using

python3 train.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan --gpu_ids 0,1,2,3 --batch_size 16 --norm instance

I have also tried to run it with and without the --norm instance parameter and
also tried with --batch_size 4. This always leads to the same result:
The program stops at "create web directory" (I've let it run for a couple of days at this point, without any noticeable progress). It looks like a single python3 process is putting a single thread under full load, none of the other python3 processes get any CPU time. Trying to kill that process also seems impossible - I have had to restart the machine every time. None of the GPUs are ever under load and barely any of their memory is used.
I am using Python 3.5.2, CUDA 9.2, pytorch 1.0, cuDNN 7.4.1. The system has four 1080ti GPUs and an AMD Ryzen Threadripper 1950X.

It also happen to me. How to solve this? Guys, i need your help!

The same issue for me, any suggestion?

[junyanz]

I haven't been able to reproduce the error on my machine. @taesungp @ssnl

[ssnl]

Could you verify that basic cuda comm primitives works on your machine? E.g., try torch.cuda.broadcast. Other primitives you can try can be found at https://pytorch.org/docs/stable/cuda.html#communication-collectives