[Prompt lookup]

[iefode]

Description:

• Implementation of Prompt lookup decoding based on continuous batching pipeline (cb_promp_lookup_impl + prompt_lookup_impl)
• Update prompt_lookup_sample to use new API
• Update statistic to make of printing more usable

Ticket:

• CVS-137987

Example of usage:

• Input: return 0;
• Result Prompt lookup:

=============================== 
Total duration, ms: 3.02267
Draft model duration, ms: 0.000724718
Main model duration, ms: 3.02195
Draft model duration, %: 0.0239761
Main model duration, %: 99.976
AVG acceptance rate, %: 10.8333
=============================== 
Request_id: 0 ||| 0 0 0 0 0 0 0 0 20 20 0 0 0 0 20 100 80 0 0 0 0 0 0 60 0 0 20 0 0 0 0 0 20 0 0 50

• Result Greedy:

=============================== 
Total duration, ms: 3.18111
Draft model duration, ms: 1.538e-06
Main model duration, ms: 3.18111
Draft model duration, %: 4.83479e-05
Main model duration, %: 100
AVG acceptance rate, %: -nan
===============================

• Speedup: 100 Generated tokens: 5.24% && 300 Generated tokens: 81% (9.42 vs 5.19)

[iefode]

@ilya-lavrenov Please take a look

[ilya-lavrenov]

is perf benefits the same between SPDA and PA impls?