Fix UnicodeDecodeError for BPE-based Models (especially GLM-4) #6357

Merged

oobabooga merged 7 commits intooobabooga:devfrom GralchemOz:main

Sep 3, 2024

+6-1

Commits on Jul 25, 2024

Commits on Aug 1, 2024

Merge pull request oobabooga#6300 from oobabooga/dev
oobabooga
authored

Commits on Aug 30, 2024

Fix UnicodeDecodeError for partial character output in BPE tokenizer

GralchemOz
authored and
GralchemOz
committed