Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ CB ][ SD ] Support streaming with using stop_strings and include_stop_strings #1382

Merged
merged 24 commits into from
Dec 20, 2024
Merged

[ CB ][ SD ] Support streaming with using stop_strings and include_stop_strings #1382

merged 24 commits into from
Dec 20, 2024

Conversation

iefode
Copy link
Contributor

@iefode iefode commented Dec 13, 2024 •
edited
Loading

Details::

  • Implement streaming with using stop_strings in CB like pipelines
  • Change stop_string_match logic to encode them only once per request
  • Do not stream tokens which are matched to the part of a stop_string (Tests was a bit changes in this case according HF does not support exclude stop_strings)

Tickets:

  • CVS-158463

@iefode iefode changed the title [ CB ][ SD ] Implement streaming with using [ CB ][ SD ] Implement streaming with using stop_strings Dec 13, 2024
@github-actions github-actions bot added category: continuous batching Continuous batching category: sampling Sampling / Decoding algorithms category: speculative decoding Speculative decoding category: samples GenAI samples labels Dec 13, 2024
@ilya-lavrenov ilya-lavrenov added this to the 2025.0 milestone Dec 13, 2024
@iefode iefode marked this pull request as draft December 16, 2024 09:38
@iefode iefode force-pushed the streaming_stop_strings branch from 72d9348 to 0e91fae Compare December 16, 2024 18:22
@github-actions github-actions bot added category: LLM LLM pipeline (stateful, static) no-match-files and removed category: samples GenAI samples labels Dec 16, 2024
@github-actions github-actions bot added the category: samples GenAI samples label Dec 17, 2024
@iefode iefode changed the title [ CB ][ SD ] Implement streaming with using stop_strings [ CB ][ SD ] Implement streaming with using stop_strings and include_stop_strings via streamer & generation handling Dec 17, 2024
@github-actions github-actions bot added the category: GenAI C++ API Changes in GenAI C++ public headers label Dec 17, 2024
@github-actions github-actions bot removed category: LLM LLM pipeline (stateful, static) category: speculative decoding Speculative decoding category: GenAI C++ API Changes in GenAI C++ public headers labels Dec 17, 2024
@iefode iefode marked this pull request as ready for review December 17, 2024 20:41
ilya-lavrenov
ilya-lavrenov reviewed Dec 18, 2024
View reviewed changes
src/cpp/src/continuous_batching_impl.cpp Outdated Show resolved Hide resolved
src/cpp/src/sampler.cpp Outdated Show resolved Hide resolved
@iefode iefode changed the title [ CB ][ SD ] Implement streaming with using stop_strings and include_stop_strings via streamer & generation handling [ CB ][ SD ] Support streaming with using stop_strings and include_stop_strings Dec 20, 2024
ilya-lavrenov
ilya-lavrenov reviewed Dec 20, 2024
View reviewed changes
samples/cpp/speculative_decoding_lm/speculative_decoding_lm.cpp Outdated Show resolved Hide resolved
samples/cpp/text_generation/greedy_causal_lm.cpp Outdated Show resolved Hide resolved
src/cpp/src/sampler.hpp Show resolved Hide resolved
src/cpp/src/sequence_group.hpp Outdated Show resolved Hide resolved
src/cpp/src/sampler.cpp Outdated Show resolved Hide resolved
src/cpp/src/sampler.cpp Outdated Show resolved Hide resolved
@github-actions github-actions bot removed the category: samples GenAI samples label Dec 20, 2024
@iefode iefode enabled auto-merge December 20, 2024 10:49
@iefode iefode added this pull request to the merge queue Dec 20, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 20, 2024
@iefode iefode added this pull request to the merge queue Dec 20, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 20, 2024
@iefode iefode added this pull request to the merge queue Dec 20, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 20, 2024
@iefode iefode enabled auto-merge December 20, 2024 18:41
@iefode iefode added this pull request to the merge queue Dec 20, 2024
Merged via the queue into openvinotoolkit:master with commit 74cdfc9 Dec 20, 2024
59 checks passed
@iefode iefode deleted the streaming_stop_strings branch December 21, 2024 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ilya-lavrenov ilya-lavrenov ilya-lavrenov approved these changes

@mzegla mzegla Awaiting requested review from mzegla

Assignees

@ilya-lavrenov ilya-lavrenov

@Wovchena Wovchena

@mzegla mzegla

Labels
category: continuous batching Continuous batching category: LLM LLM pipeline (stateful, static) category: sampling Sampling / Decoding algorithms no-match-files
Projects
None yet
Milestone
2025.0
Development

Successfully merging this pull request may close these issues.
4 participants
@iefode @ilya-lavrenov @mzegla @Wovchena