[Question] How to store/pass past-key-values?

[Justus-Jonas]

Hi all,

I was wondering whether it is possible to store and pass on past key values from long prefix prompts. Has anyone tried this with vLLM before?

Many thanks!

[YijiaZhao]

Common prefix prompts in a batch?

[Justus-Jonas]

Hi @YijiaZhao,

thanks for your reply! Yes, indeed. I saw that you also worked on this (in Issue 680). Did you find a solution?

[YijiaZhao]

I've finished yet. I think it's easy to store the past KV cache, but hard to modify the attention for postfix calculation.

[zacharyblank]

@YijiaZhao I am looking for this too. Were you able to crack it? Thanks!

[pj-ml]

Also looking for this.

[hmellor]

#2614