Parallel sampling eviction#157

Merged

sunggg merged 42 commits intooctoml:batch-servingfrom masahi:parallel-sampling-eviction

Feb 2, 2024

+498-185

Commits on Jan 11, 2024

add new model for evaluating logits over multiple queries using KV cache
masahi
committed
add test
masahi
committed
clean
masahi
committed
Only the number of past tokens is needed
masahi
committed
fix build
masahi
committed
fix
masahi
committed
correctly handle num_past_tokens > sliding_window case
masahi
committed
wip
masahi
committed
blac
masahi
committed
wip
masahi
committed
wip
masahi
committed
remove cancel call back in eviction
masahi
committed
Create MultiQueryDecodeRequest
masahi
committed
only the number of past tokens is needed
masahi
committed
wip
masahi
committed
wip
masahi
committed
wip
masahi
committed
fix
masahi
committed
wip
masahi
committed
wip
masahi
committed
wip
masahi
committed
wip
masahi
committed
working?
masahi
committed
remove dbg print
masahi
committed
multi gpu works
masahi
committed
fixed sliding window logic
masahi
committed
remove dbug print
masahi
committed
clean and fix
masahi
committed
mypy
masahi
committed

Commits on Jan 13, 2024

Commits on Jan 31, 2024