Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Summary: To 2.1 Unfortunately, I can't remove the old version of the crate. The problem is that it appears in the [public API](https://docs.rs/tiktoken-rs/latest/tiktoken_rs/struct.CoreBPE.html#method.new) of `tiktoken_rs`, which we also import. That crate has a PR to upgrade its dependency [here](zurawiki/tiktoken-rs#95) but that has not been merged. Alternatives here are: 1. Fork `tiktoken_rs`. This seems worse. 2. It's strictly speaking possible to use generics combined with some other tricks to avoid needing to name v1 of `FxHashMap` in fbsource. However, that makes that code extraordinarily hard to change and work with, and doesn't seem all that clearly motivated. If there's something else I should be doing here, all ears Reviewed By: zertosh Differential Revision: D68064854 fbshipit-source-id: bd1c96abbc0ebaa178a55826bc85c61eb35e8301
- Loading branch information