About GPU utilization

[Hzj199]

In multi-machine, multi-GPU training, the InfiniBand (IB) network shows no traffic, suggesting that avg_loss = accelerator.gather(loss).mean() might not be used.

[Hzj199]

Why is it necessary to use transform_models_if_DDP(models) after accelerator.prepare?

[Isotr0py]

transform_models_if_DDP(models) is a wrong writing made by mistake and we haven't noticed it before. This will be removed in the new PR, because it breaks the gradients sync.

I think this may also break the multi-machine communication. Maybe the new PR can fix this.

I only have two GPUs on one machine and can't test in multi-machine training. Could you check if the update works on the multi-machine training?

[Hzj199]

transform_models_if_DDP(models) is a wrong writing made by mistake and we haven't noticed it before. This will be removed in the new PR, because it breaks the gradients sync.

I think this may also break the multi-machine communication. Maybe the new PR can fix this.

I only have two GPUs on one machine and can't test in multi-machine training. Could you check if the update works on the multi-machine training?

I removed transform_models_if_DDP(models) and it now runs normally on four machines with 8 GPUSs, with GPU utilization of around 95%.