Skip to content

Commit

Permalink
[Unity] Paged KV Cache as LM Support
Browse files Browse the repository at this point in the history 
This PR introduces the PagedKVCache object to Relax runtime
for the KV cache value management in batching settings.

One test file is included. Note that this file does not contain
the test of attention function/kernel. That part will be uploaded
and tested separately.
  • Loading branch information
@MasterJH5574
MasterJH5574 committed Oct 12, 2023
1 parent b138005 commit 4a5b263
Show file tree
Hide file tree
Showing 2 changed files with 1,045 additions and 0 deletions.
Loading

0 comments on commit 4a5b263

Please sign in to comment.