[V1] Add all_token_ids attribute to Request #10135
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
| @@ -324,7 +324,7 @@ def send_to_detokenizer(self, sampled: List[Tuple[Request, int]]) -> None: | |||
| ) | |||
| for req, num_tokens in sampled: | |||
| inputs.req_ids.append(req.request_id) | |||
| if len(req.output_token_ids) == num_tokens: | |||
| if req.num_output_tokens == num_tokens: | |||
There was a problem hiding this comment.
len is supported by ConstantList, but I fixed this for clarity.
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
WoosukKwon
force-pushed
the
add-all-token-ids
branch
from
f297a7c
to
34a6635
Compare
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: OmerD <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Loc Huynh <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Sumit Dubey <[email protected]>
This PR adds the
all_token_idsto theRequestclass, which is always updated withoutput_token_ids. Havingall_token_idscan be useful because we can directly on the entire list of token ids (i.e.,prompt_token_ids + output_token_ids) without the O(n) computation every time.To make sure that the two lists are updated atomically, the PR introduces ConstantList. The constant list provides an immutable view of the list with O(1) overheads.