Skip to content

Commit

Permalink
Bump tokenizers from 0.19.0 to 0.19.1
Browse files Browse the repository at this point in the history
Bumps [tokenizers](https://github.com/huggingface/tokenizers) from 0.19.0 to 0.19.1.
- [Release notes](https://github.com/huggingface/tokenizers/releases)
- [Changelog](https://github.com/huggingface/tokenizers/blob/main/RELEASE.md)
- [Commits](huggingface/tokenizers@v0.19.0...v0.19.1)

---
updated-dependencies:
- dependency-name: tokenizers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
  • Loading branch information
dependabot[bot] authored and benbrandt committed Apr 18, 2024
1 parent 7c136a6 commit ae744ad
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 5 deletions.
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ once_cell = "1.19.0"
pulldown-cmark = { version = "0.10.2", default-features = false, optional = true }
regex = "1.10.4"
tiktoken-rs = { version = "0.5.8", optional = true }
tokenizers = { version = "0.19.0", default_features = false, features = [
tokenizers = { version = "0.19.1", default_features = false, features = [
"onig",
], optional = true }
unicode-segmentation = "1.11.0"
Expand All @@ -61,7 +61,7 @@ divan = "0.1.14"
fake = "2.9.2"
insta = { version = "1.38.0", features = ["glob", "yaml"] }
more-asserts = "0.3.1"
tokenizers = { version = "0.19.0", default-features = false, features = [
tokenizers = { version = "0.19.1", default-features = false, features = [
"onig",
"http",
] }
Expand Down
2 changes: 1 addition & 1 deletion bindings/python/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ text-splitter = { path = "../..", features = [
"tokenizers",
] }
tiktoken-rs = "0.5.8"
tokenizers = { version = "0.19.0", default_features = false, features = [
tokenizers = { version = "0.19.1", default_features = false, features = [
"onig",
] }

0 comments on commit ae744ad

Please sign in to comment.