how i can use batch to accelerate

[jlygit]

hello!
I infer one image with resolution of 1280* 720 using openvino and the infer time is 129ms .
I try to crop this image into four patchs with resolution of 640* 360, and infer these four batches, and the infer time is about 129ms .
Why it has not been accelerated when i using batching? how i can use batch to accelerate or other way?

Thanks,
jly

[elvin-n]

The performance boost from batch happens due to more efficient parallelization of bigger amount of work vs less amount of work. In regular case when you increase batch, you can just load computing capacity the more efficient way. I.e. the throughput will be increased (whlie latency will be increased from batch too, that is important to understand trade-off of batch mode).

129ms - means that network is quite heavy and probably even in case of bigger picture in your scenario, all cores are loaded the most efficient way. And the same throughput says that parallelization on spatial dimention has the same efficency as parallelization on batch dimention.

[dkurt]

@jlygit, could you please share which model is used? Is it from https://github.com/opencv/open_model_zoo?

[jlygit]

@dkurt Sorry, I can't share it since we used our own model.

[dkurt]

@jlygit, Is it object detection model or not? If so, try to run it on smaller input (use net's reshape to fit the shapes before). Then try to find the resolution which still gives satisfying accuracy.

[zhaojw219]

How can I use batch in python demo?Please help me.Thanks!