Why does it take more time when I use two models at he same time?

[jlygit]

I created a InferRequest using model A, and the infer time is 60ms,also another InferRequest using model B, and the infer time is 43ms.
However, I created two InferRequest using Model A (called infer_a) and Model B (called infer_b) at the same time and infer the same image sequentially using infer_a, infer_b, it takes time 105ms and 264ms respectively.
Why does it take more time when I use two models?

[elvin-n]

I suppose you are talking about inference on CPU.
This should be a problem with oversubscriptions in inference engine based on OpenMP. There are two solutions for this issue:

1. Follow the optimization guide: https://software.intel.com/en-us/articles/OpenVINO-Inference-Engine-Optimization-Guide , section "CPU Checklist" and "Performance Aspects of Running Multiple Requests Simultaneously"

or

2. Recompile inference engine with TBB

[jlygit]

@elvin-nnov
it works, thanks very much!

[jlygit]

@elvin-nnov
hello,
How can i recompile inference engine with TBB? Is any way here?
thanks.

[emptyVoid]

It's pretty straightforward: you just add -DTHREADING=TBB to cmake arguments:

cmake -DCMAKE_BUILD_TYPE=Release -DTHREADING=TBB ..
make -j16