Regarding multi-gpu training

[YibinXie]

Q1: I use normal dataparallel training with this command but got the error.

cd ..
python tools/train.py configs/top_down/hrnet/coco/hrnet_w32_coco_256x192.py --gpus 4

Q2: I use distributed training with this command but got the error.

cd ..
bash tools/dist_train.sh configs/top_down/hrnet/coco/hrnet_w32_coco_256x192.py 4

[innerlee]

Can single gpu be trained succefully?

[YibinXie]

Can single gpu be trained succefully?

Yes, single gpu works fine.

[innerlee]

For multiple GPUs, the distributed version is recommended.

As for the error, you can set num_workers to 0 in the config and try again

[YibinXie]

For multiple GPUs, the distributed version is recommended.

As for the error, you can set num_workers to 0 in the config and try again

I set workers to 0 and It's stuck here.
I never used distributed training before. I think your code is right. Yesterday, I tried to learn how to use pytorch distributed training, and I managed to distributed training a classification task on a very small dataset. But when I use the same way to train COCO dataset for pose estimation. It failed. I think maybe something goes wrong with my machine.

[innerlee]

What's the version of pytorch? Could you try an older version, say 1.3?

[YibinXie]

What's the version of pytorch? Could you try an older version, say 1.3?

My version is 1.6.0, cause I want to use auto mixed precision training.
Ok, I will try an older version later.
Many thanks for your instant reply.

[YibinXie]

What's the version of pytorch? Could you try an older version, say 1.3?

I've tried to reduce my PyTorch to an older version 1.3.0 and 1.4.0, unfortunately, these two versions don't make it in my cluster. My assumption something goes wrong with this cluster (4 M40 24GB). But I didn't figure out the reasons.

My M40 cluster info:
24 Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
120G memory
4*Tesla M40 24GB
I list the info above, maybe you could give some comments.

Then I clone the codes to another machine with 2 TITAN RTX 24GB with PyTorch 1.6.0. And it works fine with distributed training.

[innerlee]

It is nearly impossible to diagnose cluster issues without access to it. Maybe you can find IT to help?

[YibinXie]

It is nearly impossible to diagnose cluster issues without access to it. Maybe you can find IT to help?

Sure, it works anyway. Thanks for your reply.
By the way, have you thought to add the PyTorch amp feature to the project?

[innerlee]

We have some fp16 support in progress #200. PyTorch amp is one way to do it, and it might be included in near future.

[innerlee]

You may post a request here #9 to help us prioritize

[YibinXie]

You may post a request here #9 to help us prioritize

Ok, thanks.

[MrRaptorious]

Hey, do you know if i can run multiple GPUs on one system, without using the distributed approach?
I do not have access to network stuff and therefore cant run the train_dist script.

Does MMPose even support that? I checked the runner.py in MMEngine and it seems like there is no multi GPU training without opening ports or something like that.