Skip to content

Commit

Permalink
[Unity] Paged KV Cache as LM Support
Browse files Browse the repository at this point in the history 
This PR introduces the PagedKVCache object to Relax runtime
for the KV cache value management in batching settings.

One test file is included. Note that this file does not contain
the test of attention function/kernel. That part will be uploaded
and tested separately.
  • Loading branch information
@MasterJH5574
MasterJH5574 committed Oct 12, 2023
1 parent b138005 commit 8ef57ba
Show file tree
Hide file tree
Showing 2 changed files with 1,082 additions and 0 deletions.
Loading

0 comments on commit 8ef57ba

Please sign in to comment.