[Nano] OpenVINO API for Async Inference

[zhentaocc]

Problem

Nano has enabled OpenVINOModel.forward with sync mode. For example:

from bigdl.nano.openvino import OpenVINOModel
ov_model = OpenVINOModel("model.xml")
for img in dataset:
    out = ov_model(img)

This will get the results batch by batch. As for async mode in https://docs.openvino.ai/latest/openvino_docs_OV_UG_Python_API_exclusives.html#asyncinferqueue, we didn't enable this for Nano. With async inference, we can obtain better performance.

API Design

For raw OpenVINO:

from bigdl.nano.openvino import OpenVINOModel
ov_model = OpenVINOModel("model.xml")
model.predict(dataset: np_array)

OpenVINOModel.precit(...) will be the function to enable async inference loop.

For Pytorch OpenVINO"

from bigdl.nano.pytorch.trainer import Trainer
trainer = Trainer()
ov_model = Trainer.trace(pytorch_model, accelerator='openvino')
trainer.predict(ov_model, dataloader)

Trainer.predict() will be overridden to run async inference loop.

Tasks

assigned to @hjzin

1. enable async API for raw OpenVINO
2. enable async API for Pytorch OpenVINO
3. modify [Nano] Openvino quantization notebooks with nano #5491

[jason-dai]

Why do we need two different sets of APIs? Make it consistent for this specific feature.

[zhentaocc]

Why do we need two different sets of APIs? Make it consistent for this specific feature.

Because they are for different frameworks actually.
There is no Trainer for raw OpenVINO since we don't require Pytorch-Lightning installation in this case.
For pytorch-lightning:

trainer.predict(ov_model, dataloader)

For Keras:

ov_model.predict(dataloader)

For OpenVINO without pytorch/tf:

ov_model.predict(data)

They seem to be kind of consistent.

[jason-dai]

I think we can then have all cases to use model.predict, no? And model.predict can support both sync and async modes.

[zhentaocc]

I think we can then have all cases to use model.predict, no?

Sure we can do this. This will introduce a new API for Pytorch users, model.predict(...). Shall we make trainer.predict same as model.predict(...)? Users might be confused with these two API with the same name.

And model.predict can support both sync and async modes.

Is it necessary to support sync mode? seems that async mode can always obtain better performance in this case.

[jason-dai]

I think we can then have all cases to use model.predict, no?

Sure we can do this. This will introduce a new API for Pytorch users, model.predict(...). Shall we make trainer.predict same as model.predict(...)? Users might be confused with these two API with the same name.

And model.predict can support both sync and async modes.

Is it necessary to support sync mode? seems that async mode can always obtain better performance in this case.

1. No need to add or update trainer.predict
2. Change to model.async_predict?