Inference and training time is far slower than Caffe2?

[Tangshitao]

I test the inference and training time per batch. The configuration is the same as Caffe2's version of detectron. The result is as follows:
model | inference time | train time
Res-50-fpn | 0.182s+0.006s | 1.013s/iter
This speed is far behind the one claimed in the official detectron tested under the same environment. What do you think the problem might be?

[gdshen]

I test Res-50-C4, and it gives 0.18s/im, much the same as the one claims in Detectron

[Tangshitao]

How many gpus do you use for inference? My results show the speed is almost the same as the official one when using 1 gpu and much slower when using 8 gpus.

[happyharrycn]

I have to point out an issue when comparing run-time performance for multi-gpu training. The Caffe2 timing assumes peer-to-peer access across all GPUs (see their issue #32). This is typically not the case on many GPU servers, where the training will slow down due to extra communication cost.