Skip to content

Commit

Permalink
Add C++ example to README (#320)
Browse files Browse the repository at this point in the history
  • Loading branch information
helena-intel authored Nov 20, 2024
1 parent ada161f commit aeb1bc2
Showing 1 changed file with 53 additions and 0 deletions.
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -431,6 +431,59 @@ print(tokenized := tokenizer(["Test string"])["input_ids"]) # [[24235 47429]]
print(detokenizer(tokenized)["string_output"]) # ['Test string']
```
### C++ Usage example
This example shows how to run inference with C++ on a text-classification model from Hugging Face. It
expects the path to a model directory as parameter, and prints the logits returned by the model inference.
Export an example model by running the following command after `pip install optimum[openvino]`:
```sh
optimum-cli export openvino microsoft/deberta-base-mnli deberta-base-mnli-ov
```
```cpp
#include <openvino/openvino.hpp>
#include <iostream>
#include <filesystem>
int main(int argc, char* argv[]) {
std::string dirname = argv[1];
std::filesystem::path dir_path(dirname);
std::filesystem::path model_xml = dir_path / "openvino_model.xml";
std::filesystem::path tokenizer_xml = dir_path / "openvino_tokenizer.xml";
ov::Core core;
// use "openvino_tokenizers.dll" on Windows, "libopenvino_tokenizers.dylib" on macOS
core.add_extension("libopenvino_tokenizers.so");
ov::InferRequest tokenizer_request = core.compile_model(tokenizer_xml, "CPU").create_infer_request();
std::string prompt="Hello world!";
tokenizer_request.set_input_tensor(ov::Tensor{ov::element::string, {1}, &prompt});
tokenizer_request.infer();
ov::Tensor input_ids = tokenizer_request.get_tensor("input_ids");
ov::Tensor attention_mask = tokenizer_request.get_tensor("attention_mask");
ov::InferRequest infer_request = core.compile_model(model_xml, "CPU").create_infer_request();
infer_request.set_tensor("input_ids", input_ids);
infer_request.set_tensor("attention_mask", attention_mask);
infer_request.infer();
auto output = infer_request.get_tensor("logits");
const float *output_buffer = output.data<const float>();
size_t num_elements = output.get_size();
for (size_t i = 0; i < num_elements; i++) {
std::cout << output_buffer[i] << " ";
}
std::cout << std::endl;
return 0;
}
```
## Supported Tokenizer Types
| Huggingface <br/>Tokenizer Type | Tokenizer Model Type | Tokenizer | Detokenizer |
Expand Down

0 comments on commit aeb1bc2

Please sign in to comment.