add generation time metrics

[pavel-esir]

• Added performance metrics and updated Readme with description how to use them
• Added cpp and python sample for benchmarking

Sample to calculate and visualize performance metrics.

import openvino_genai as ov_genai
import tqdm
import pandas as pd
import matplotlib.pylab as pl

pipe = ov_genai.LLMPipeline('TinyLlama-1.1B-Chat-v1.0/')
config = ov_genai.GenerationConfig(max_new_tokens=15)
metrics_df = pd.DataFrame(columns=['batch_size', 'throughput', 'ttft', 'tpot', 'std_throughput', 'std_ttft', 'std_tpot'])

num_iter = 3
for batch_size in tqdm.tqdm([1, 2, 4, 16, 32, 64, 128]):
    prompts = ["The Sky is blue because"] * batch_size
    res = pipe.generate(prompts, config)
    metrics = res.perf_metrics
    
    for _ in range(num_iter - 1):
        res = pipe.generate(prompts, config)
        metrics += res.perf_metrics
    metrics_df = metrics_df._append({
        'throughput': metrics.get_throughput().mean, 'ttft': metrics.get_ttft().mean, 'tpot': metrics.get_tpot().mean,
        'std_throughput': metrics.get_throughput().std, 'std_ttft': metrics.get_ttft().std, 'std_tpot': metrics.get_tpot().std,
        'batch_size': batch_size, 
    }, ignore_index=True)

fig, axes = pl.subplots(nrows=3, ncols=1, figsize=(6, 8), sharex=True)

axes[0].plot(metrics_df['batch_size'], metrics_df['throughput'], '-o')
axes[1].plot(metrics_df['batch_size'], metrics_df['ttft'], '-o', )
axes[2].plot(metrics_df['batch_size'], metrics_df['tpot'], '-o')

axes[0].set_ylabel('Throughput'), axes[1].set_ylabel('TTFT'), axes[2].set_ylabel('TPOT')
axes[2].set_xlabel('Batch Size')
axes[0].grid(True), axes[1].grid(True), axes[2].grid(True)
pl.tight_layout()

ticket: CVS-132859

[Wovchena]

Don't cast duration until you really need to use the value (for printing). Return duration itself. That ensures the best accuracy. When you divide a duration, change it's representation to float or doulbe. For example https://github.com/openvinotoolkit/openvino/blob/ffc135cb1240831411799bdb82ecac352c956f22/samples/cpp/benchmark/throughput_benchmark/main.cpp#L19. But your implementation needs an extra step: when mean is computed keep using source units (most likely nanoseconds, but it's unspecified, you can't rely on that) with float or double representation. Cast the duration to Ms or any other suitable unit to use count() and print.

[pavel-esir]

Done. Now durations are stored in microsecond chrono::duration<float, std::ratio<1, 1000000>> for better accuracy. If i store them in ms then tokenization/detokenization times can sometimes be 0, with microseconds.

I convert them only when mean/std are calculated.

[Wovchena]

You shouldn't store duration as any of the specified units. Store whatever time_point_one - time_point_zero returns. There's also a cast to int representation in constructors. For example

    auto start_time = std::chrono::steady_clock::now();
    m_pimpl = std::make_unique<StatefulLLMPipeline>(request, tokenizer, generation_config);
    auto stop_time = std::chrono::steady_clock::now();
    m_pimpl->m_load_time_ms = std::chrono::duration_cast<std::chrono::milliseconds>(stop_time - start_time).count();   

Which is bad in two ways: the first is described above, the second is that the used int representation in the cast. That's going to divide by 1000, because the default type is usually nanoseconds.

[pavel-esir]

now PR is final. @Wovchena please take a look

[Wovchena]

It got a conflict

[pavel-esir]

It got a conflict

Resolved. Metrics match to llm_bench's numbers. Will open a separate PR to switch to native counters.

[Wovchena]

When it gets merged, please, open another PR adding it to C++ and Python docstrings.

[pavel-esir]

Done #713

[Wovchena]

evaluate_statistics() is called on every +. Given that it happens during benchmarking loop and most of the results are thrown away, it's worth providing a getter or a standalone function to do that job.

[pavel-esir]

Added getters. Now in order to get user should call perf_metrics.get_tokenization_duration().mean if statistics are fresh and already evaluated then it will return values, if they are not fresh getter will call evaluate_statistics()

Comments were applied