Skip to content

Commit

Permalink
Bump minimum python version to 3.8
Browse files Browse the repository at this point in the history
  • Loading branch information
benbrandt committed Dec 27, 2023
1 parent 73152f6 commit ab7121b
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 2 deletions.
1 change: 1 addition & 0 deletions bindings/python/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

### Breaking Changes

- Minimum Python version is now 3.8.
- Due to using binary search, there are some slight differences at the edges of chunks where the algorithm was a little greedier before. If two candidates would tokenize to the same amount of tokens that fit within the capacity, it will now choose the shorter text. Due to the nature of of tokenizers, this happens more often with whitespace at the end of a chunk, and rarely effects users who have set `trim_chunks=true`. It is a tradeoff, but would have made the binary search code much more complicated to keep the exact same behavior.

## v0.2.4
Expand Down
2 changes: 1 addition & 1 deletion bindings/python/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ name = "semantic_text_splitter"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.20.0", features = ["abi3-py37"] }
pyo3 = { version = "0.20.0", features = ["abi3-py38"] }
text-splitter = { version = "0.5.0", features = ["tiktoken-rs", "tokenizers"] }
tiktoken-rs = "0.5.8"
tokenizers = { version = "0.15.0", default_features = false, features = [
Expand Down
2 changes: 1 addition & 1 deletion bindings/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "maturin"

[project]
name = "semantic-text-splitter"
requires-python = ">=3.7"
requires-python = ">=3.8"
classifiers = [
"Programming Language :: Rust",
"Programming Language :: Python :: Implementation :: CPython",
Expand Down

0 comments on commit ab7121b

Please sign in to comment.