Prefix caching improvements #758
Conversation
| namespace ov { | ||
| namespace genai { | ||
| size_t Sequence::_make_hash(size_t content_length) { | ||
| auto sequence_group = get_sequence_group_ptr(); |
There was a problem hiding this comment.
do we need to add an assert that content_length corresponds to last uncomputed block? E.g. if we have 1 block with has, but content_length is 3x of block_size.
E.g. block_start_idx / block_size == m_prefix_hashes.size()
There was a problem hiding this comment.
There is a case when block_start_idx / block_size < m_prefix_hashes.size().
When we restore blocks of prompt first we check hash of full block and it is saved in m_prefix_hashes, then if we couldn't find hash of full block in cashed_blocks we check hashes of partially completed content of this block. So content_length in this case is less than m_prefix_hashes.size() * block_size.
So I added assert block_start_idx / block_size <= m_prefix_hashes.size().
Applied comments from #675