Skip to content

Commit

Permalink
Apply suggestions from code review
Browse files Browse the repository at this point in the history
Co-authored-by: Yaroslav Tarkan <[email protected]>
Co-authored-by: Xiake Sun <[email protected]>
Co-authored-by: Ilya Lavrenov <[email protected]>
  • Loading branch information
4 people authored May 24, 2024
1 parent 70f1177 commit da729ba
Show file tree
Hide file tree
Showing 4 changed files with 11 additions and 12 deletions.
12 changes: 6 additions & 6 deletions src/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --weigh
pip install openvino-genai
```

LLMPipeline is the main object used for decoding. You can initiliza it straigh away from the folder with the converted model. It will automanically load the main model, tokenizer, detokenizer and default generation configuration.
`LLMPipeline` is the main object used for decoding. You can initialize it straight away from the folder with the converted model. It will automatically load the main model, tokenizer, detokenizer and default generation configuration.

### Python

Expand Down Expand Up @@ -129,7 +129,7 @@ int main(int argc, char* argv[]) {
for (size_t i = 0; i < questions.size(); i++) {
std::cout << "question:\n";
cout << prompt << endl;
std::cout << prompt << std::endl;
auto answer = pipe(prompt, config, streamer);
// no need to print answer, streamer will do that
Expand All @@ -138,7 +138,7 @@ int main(int argc, char* argv[]) {
}
```

Streaming exapmle with lambda function
Streaming example with lambda function

``` cpp

Expand All @@ -156,11 +156,11 @@ int main(int argc, char* argv[]) {
Streaming with custom class
``` cpp
#include <streamer_base.hpp>
#include "openvino/genai/streamer_base.hpp"
#include "openvino/genai/llm_pipeline.hpp"
#include <iostream>
class CustomStreamer: publict StreamerBase {
class CustomStreamer: public ov::StreamerBase {
public:
void put(int64_t token) {
/* custom decoding/tokens processing code
Expand All @@ -180,6 +180,6 @@ int main(int argc, char* argv[]) {
std::string model_path = argv[1];
ov::LLMPipeline pipe(model_path, "CPU");
cout << pipe.generate("The Sun is yellow bacause", custom_streamer);
std::cout << pipe.generate("The Sun is yellow bacause", custom_streamer);
}
```
3 changes: 1 addition & 2 deletions src/cpp/include/openvino/genai/generation_config.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,11 @@ enum class StopCriteria { early, heuristic, never };
* @param eos_token_id id of <eos> token
* @param bos_token <bos> token string representation
* @param eos_token <eos> token string representation
* @param draft_model draft model for assitive decoding
*/
class OPENVINO_GENAI_EXPORTS GenerationConfig {
public:
GenerationConfig() = default;
GenerationConfig(std::string json_path);
explicit GenerationConfig(std::string json_path);

// Generic
size_t max_new_tokens = SIZE_MAX;
Expand Down
6 changes: 3 additions & 3 deletions src/cpp/include/openvino/genai/llm_pipeline.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#include <optional>
#include <variant>

#include <openvino/core/any.hpp>
#include "openvino/core/any.hpp"
#include "openvino/genai/generation_config.hpp"
#include "openvino/genai/tokenizer.hpp"
#include "openvino/genai/streamer_base.hpp"
Expand Down Expand Up @@ -174,10 +174,10 @@ class OPENVINO_GENAI_EXPORTS LLMPipeline {
};

/*
* utils that allow to use generate and operarator() in the folllowing way:
* utils that allow to use generate and operator() in the following way:
* pipe.generate(input_ids, ov::max_new_tokens(200), ov::temperature(1.0f),...)
* pipe(text, ov::max_new_tokens(200), ov::temperature(1.0f),...)
* All names match to names in cofnig except streamer.
* All names match to names in config except streamer.
*/
static constexpr ov::Property<size_t> max_new_tokens{"max_new_tokens"};
static constexpr ov::Property<size_t> max_length{"max_length"};
Expand Down
2 changes: 1 addition & 1 deletion src/cpp/include/openvino/genai/streamer_base.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ namespace ov {
class StreamerBase {
public:
Tokenizer m_tokenizer;
StreamerBase(Tokenizer tokenizer): m_tokenizer(tokenizer) {};
explicit StreamerBase(Tokenizer tokenizer): m_tokenizer(tokenizer) {}
StreamerBase() = default;

/// @brief put is called every time new token is decoded
Expand Down

0 comments on commit da729ba

Please sign in to comment.