Code

Use binary search in TextChunks#71

Code

Merged

benbrandt merged 8 commits intomainbenbrandt/text-splitter:mainfrom binary-searchbenbrandt/text-splitter:binary-searchCopy head branch name to clipboardDec 27, 2023

+19,765-17,811

Commits

Commits on Dec 26, 2023

• Use a binary search to fill TextChunks

  Show description for 1b7a352
  

  benbrandt

  and

  bradfier

  committed
  1b7a352

Commits on Dec 27, 2023

• Update snapshots for binary search

  benbrandt

  committed
  2550fa0

• Limit next section based on the encoded offsets to limit search space

  benbrandt

  committed
  12c6cac

• Use iterator based approach for chunk size to avoid extra allocations

  benbrandt

  committed
  294c1bd

• remove unneeded flag in regex matches

  benbrandt

  committed
  4b0effc

• Update changelog, bump version, and use higher version of tiktoken

  benbrandt

  committed
  56db887

• Bump required versions of both tokenizer crates

  benbrandt

  committed
  93a0272

• add back onig feature for tokenizers

  benbrandt

  committed
  be5f280