Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#5043 - Ability to specify token breaking zones when calling tokenizer #5044

Merged
merged 2 commits into from
Sep 7, 2024
Merged

#5043 - Ability to specify token breaking zones when calling tokenizer #5044

merged 2 commits into from
Sep 7, 2024

Conversation

reckart
Copy link
Member

@reckart reckart commented Sep 7, 2024

What's in the PR

  • Added new signature to the tokenizer call
  • Added test
  • Consolicated existing code

How to test manually

  • No specific test procedure

Automatic testing

  • PR includes unit tests

Documentation

  • PR updates documentation

@reckart
#5043 - Ability to specify token breaking zones when calling tokenizer
41b137d 
- Added new signature to the tokenizer call
- Added test
- Consolicated existing code
@reckart reckart added the ⭐️ Enhancement New feature or request label Sep 7, 2024
@reckart reckart added this to the 34.0 milestone Sep 7, 2024
@reckart reckart self-assigned this Sep 7, 2024
@reckart reckart merged commit a63eacc into main Sep 7, 2024
3 checks passed
@reckart reckart deleted the feature/5043-Ability-to-specify-token-breaking-zones-when-calling-tokenizer branch September 7, 2024 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@reckart reckart

Labels
⭐️ Enhancement New feature or request
Projects
Kanban
Archived in project
Milestone
34.0
Development

Successfully merging this pull request may close these issues.
1 participant
@reckart