Skip to content

Commit

Permalink
[Unity] Paged KV Cache as LM Support
Browse files Browse the repository at this point in the history 
This PR introduces the PagedKVCache object to `lm_support.cc`
for the KV cache value management in batching settings.

One test file is included. Note that this file does not contain
the test of attention function/kernel. That part will be uploaded
and tested separately.
  • Loading branch information
@MasterJH5574
MasterJH5574 committed Oct 10, 2023
1 parent b138005 commit 7671ddb
Show file tree
Hide file tree
Showing 2 changed files with 831 additions and 0 deletions.
Loading

0 comments on commit 7671ddb

Please sign in to comment.