[C++] How to predict fast?

[bitnick10]

for (int i = 0; i < 1000*1000*1000; i++) {
    auto &input_tensor = input_tensor_vec[i];
    auto output_tensors = session_->Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1);
}

This is really slow,is there some method make this faster (using C++)?

[hariharans29]

Your model must support "batching" inherently and then you can feed in a "batched" input.

For example, if your model has graph input (taking image case) of shape [1, 3, 224, 224] - it can only take one 3 channel image of height and width 224. But if it has graph input like this - ["some_string", 3, 224, 224] - it can handle "batched" input of any size.

Hope this helps.

[bitnick10]

import onnx
mp = onnx.load_model('aa.onnx')
mp.graph.input[0].type.tensor_type.shape.dim[0].dim_param = 'None'
onnx.save(mp, 'aa_batch.onnx')

[C++] after session_ load aa_batch.onnx.
auto output_tensors = session_->Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1);
It runs very fast! Thanks @hariharans29

[cena001plus]

import onnx
mp = onnx.load_model('aa.onnx')
mp.graph.input[0].type.tensor_type.shape.dim[0].dim_param = 'None'
onnx.save(mp, 'aa_batch.onnx')

[C++] after session_ load aa_batch.onnx. auto output_tensors = session_->Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1); It runs very fast! Thanks @hariharans29

Can it also speed up by simply modifying the model to dynamic input, without modifying the output node, and not batching the input data? Hope to hear from you.