OpenVINO-ExecutionProvider.md

OpenVINO Execution Provider

OpenVINO Execution Provider enables deep learning inference on Intel CPUs, Intel integrated GPUs and Intel^® Movidius^TM Vision Processing Units (VPUs). Please refer to this page for details on the Intel hardware supported.

Build

For build instructions, please see the BUILD page.

ONNX Layers supported using OpenVINO

The table below shows the ONNX layers supported using OpenVINO Execution Provider and the mapping between ONNX layers and OpenVINO layers. The below table also lists the Intel hardware support for each of the layers. CPU refers to Intel^® Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. VPU refers to USB based Intel^® Movidius^TM VPUs as well as Intel^® Vision accelerator Design with Intel Movidius ^TM MyriadX VPU. FPGA refers to Intel^® Vision Accelerator Design with an Intel^® Arria^® 10 FPGA.

ONNX Layers	OpenVINO Layers	CPU	GPU	VPU	FPGA
Add	Eltwise (operation=sum)	Yes	Yes	Yes	Yes
AveragePool	Pooling(pool_method=avg)	Yes	Yes	Yes	Yes
BatchNormalization	Scaleshift (can be fused into Convlution or Fully Connected)	Yes	Yes	Yes	Yes
Concat	Concat	Yes	Yes	Yes	Yes
Conv	Convolution	Yes	Yes	Yes	Yes
Div	Eltwise(operation = mul)->Power	Yes	Yes	Yes	Yes
Dropout	Ignored	Yes	Yes	Yes	Yes
Flatten	Reshape	Yes	Yes	Yes	No
Gemm	FullyConnected	Yes	Yes	Yes	Yes
GlobalAveragePool	Pooling	Yes	Yes	Yes	Yes
GlobalMaxPool	Pooling	Yes	Yes	Yes	Yes
Identity	Ignored	Yes	Yes	Yes	Yes
ImageScaler	ScaleShift	Yes	Yes	Yes	Yes
LRN	Norm	Yes	Yes	Yes	Yes
MatMul	FullyConnected	Yes	Yes*	No	Yes
MaxPool	Pooling(pool_method=max)	Yes	Yes	Yes	Yes
Mul	Eltwise(operation=mul)	Yes	Yes	Yes	No
Relu	ReLU	Yes	Yes	Yes	Yes
Reshape	Reshape	Yes	Yes	Yes	No
Softmax	SoftMax	Yes	Yes	Yes	No
Sum	Eltwise(operation=sum)	Yes	Yes	Yes	Yes
Sub	Power->Eltwise(operation = sum)	Yes	Yes	Yes	Yes
Transpose	Permute	Yes	Yes	Yes	No
UnSqueeze	Reshape	Yes	Yes	Yes	No
LeakyRelu	ReLU	Yes	Yes	Yes	Yes

*MatMul is supported in GPU only when the following layer is an Add layer in the topology.

Topology Support

Below topologies are supported from ONNX open model zoo using OpenVINO Execution Provider

Image Classification Networks

Topology	CPU	GPU	VPU	FPGA
bvlc_alexnet	Yes	Yes	Yes	Yes***
bvlc_googlenet	Yes	Yes	Yes	Yes***
bvlc_reference_caffenet	Yes	Yes	Yes	Yes***
bvlc_reference_rcnn_ilsvrc13	Yes	Yes	Yes	Yes***
densenet121	Yes	Yes	Yes	Yes***
Inception_v1	Yes	Yes	Yes**	Yes***
Inception_v2	Yes	Yes	Yes	Yes***
Shufflenet	Yes	Yes	Yes	Yes***
Zfnet512	Yes	Yes	Yes	Yes***
Squeeznet 1.1	Yes	Yes	Yes	Yes***
Resnet18v1	Yes	Yes	Yes	Yes***
Resnet34v1	Yes	Yes	Yes	Yes***
Resnet50v1	Yes	Yes	Yes	Yes***
Resnet101v1	Yes	Yes	Yes	Yes***
Resnet152v1	Yes	Yes	Yes	Yes***
Resnet18v2	Yes	Yes	Yes	Yes***
Resnet34v2	Yes	Yes	Yes	Yes***
Resnet50v2	Yes	Yes	Yes	Yes***
Resnet101v2	Yes	Yes	Yes	Yes***
Resnet152v2	Yes	Yes	Yes	Yes***
Mobilenetv2	Yes	Yes	Yes	Yes***
vgg16	Yes	Yes	Yes	Yes***
vgg19	Yes	Yes	Yes	Yes***

Image Recognition Networks

Topology	CPU	GPU	VPU	FPGA
MNIST	Yes	Yes	Yes	Yes***

Object Detection Networks

Topology	CPU	GPU	VPU	FPGA
TinyYOLOv2	Yes	Yes	Yes	Yes***
ResNet101_DUC_HDC	Yes	No	No	Yes***

***FPGA only runs in HETERO mode wherein the layers that are not supported on FPGA fall back to OpenVINO CPU.

Application code changes for VAD-M performance scaling

VAD-M has 8 VPUs and is suitable for applications that require multiple inferences to run in parallel. We use batching approach for performance scaling on VAD-M.

Below python code snippets provide sample classification code to batch input images, load a model and process the output results.

import onnxruntime as rt
from onnxruntime import get_device
import os
import os.path
import sys
import cv2
import numpy
import time
import glob

Load the input onnx model

sess = rt.InferenceSession(str(sys.argv[1]))
print("\n")

Preprocessing input images

for i in range(iters):
   y = None
   images = [cv2.imread(file) for file in glob.glob(str(sys.argv[2])+'/*.jpg')]
   for img in images:
     # resizing the image
     img = cv2.resize(img, (224,224))
     # convert image to numpy
     x = numpy.asarray(img).astype(numpy.float32)
     x = numpy.transpose(x, (2,0,1))
     # expand the dimension and batch the images
     x = numpy.expand_dims(x,axis=0)
     if y is None:
        y = x
     else:
        y = numpy.concatenate((y,x), axis=0)

Start Inference

   res = sess.run([sess.get_outputs()[0].name], {sess.get_inputs()[0].name: y})

Post-processing output results

   print("Output probabilities:")
   i = 0
   for k in range(batch_size):
       for prob in res[0][k][0]:
          print("%d : %7.4f" % (i, prob))