Skip to content

Commit

Permalink
Merge pull request #10 from hasktorch/fix/unsafe
Browse files Browse the repository at this point in the history
Allow invalid_reference_casting
  • Loading branch information
junjihashimoto authored Apr 21, 2024
2 parents f6e9f98 + 23944f1 commit addbd19
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions tokenizers/src/models/bpe/trainer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,7 @@ impl BpeTrainer {
word_counts: &HashMap<String, u32>,
model: &mut BPE,
) -> Result<Vec<AddedToken>> {
#![allow(invalid_reference_casting)]
let mut word_to_id: HashMap<String, u32> = HashMap::with_capacity(self.vocab_size);
let mut id_to_word: Vec<String> = Vec::with_capacity(self.vocab_size);

Expand Down

0 comments on commit addbd19

Please sign in to comment.