Nano: trace to openvino inference engine need a thread(core) num control option

[TheaperDeng]

For onnxruntime, users could control how many cores their accelerated model could use, while we don't have such option for openvino right now.

[rnwang04]

Currently, no matter what the number of intra_op_num_threads and inter_op_num_threads is, only one core is used.
And more threads, inference is slower.

[jason-dai]

Currently, no matter what the number of intra_op_num_threads and inter_op_num_threads is, only one core is used. And more threads, inference is slower.

@qiyuangong Is this behavior expected?

[TheaperDeng]

Currently, no matter what the number of intra_op_num_threads and inter_op_num_threads is, only one core is used. And more threads, inference is slower.

@qiyuangong Is this behavior expected?

After huge amount of experiment and feedback from team members and customers I found

ONNXRuntime's core num control is completely fragile and useless, it may not respect user's input randomly

For fp32:

• some hardware (e.g. i7-9700), the control is stable and follows the setting
• some hardware (e.g. cpx), the control is useless, and the core usage might change randomly even on same SKU.

For int8

• some hardware (e.g. i7-9700), no matter how many cores you set, it will only use 1 core
• some hardware (e.g. icx), it will use 16 cores if you set < 16 cores, and respect your input if you set > 16 cores

We are contacting onnxruntime support team (MS and Intel) to find some solution.

[rnwang04]

After unset KMP_AFFINITY, onnxruntimie works fine.