Skip to content

Commit

Permalink
Fix markdown table formatting
Browse files Browse the repository at this point in the history
  • Loading branch information
benbrandt committed Mar 25, 2024
1 parent 9ba3da5 commit ab62a2b
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,10 +188,10 @@ There are lots of methods of determining sentence breaks, all to varying degrees

### Tokenizer Support

| Dependency Feature | Version Supported | Description |
| ------------------ | ----------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tiktoken-rs` | `0.5.8` | Enables `[Text | Markdown]Splitter::new`to take`tiktoken_rs::CoreBPE` as an argument. This is useful for splitting text for OpenAI models. |
| `tokenizers` | `0.15.2` | Enables `[Text | Markdown]Splitter::new`to take`tokenizers::Tokenizer` as an argument. This is useful for splitting text models that have a Hugging Face-compatible tokenizer. |
| Dependency Feature | Version Supported | Description |
| ------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tiktoken-rs` | `0.5.8` | Enables `(Text/Markdown)Splitter::new` to take `tiktoken_rs::CoreBPE` as an argument. This is useful for splitting text for OpenAI models. |
| `tokenizers` | `0.15.2` | Enables `(Text/Markdown)Splitter::new` to take `tokenizers::Tokenizer` as an argument. This is useful for splitting text models that have a Hugging Face-compatible tokenizer. |

## Inspiration

Expand Down

0 comments on commit ab62a2b

Please sign in to comment.