Multi GPU RuntimeError: Expected device cuda:0 but got device cuda:7

[zidanexu]

hi @glenn-jocher
I try to reproduce training result.

using command above , 8 GPU Tela P40.
when finish 1 epoch training., The test process broken.

[github-actions]

Hello @zidanexu, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Google Colab Notebook, Docker Image, and GCP Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI surveillance systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@zidanexu thank you for your bug report. We can successfully reproduce this issue. It appears to be caused by self.grid, a Detect() layer list, which is sent to a device during training. It is not transferred like normal parameters/buffers because it is not in the layer buffer list as it is a list rather than a tensor. We will look into this.

[glenn-jocher]

Fix complete. git pull to receive fix.

[lucasjinreal]

this error still exists on multi GPU training. on pytorch 1.5

RuntimeError: Model replicas must have an equal number of parameters.

it seems not fix yet, any ideas?