How to Read Information in Large Tokenizer's Vocabulary

[kaizhuanren]

TLDR; This is how the byte-level BPE works. Main advantages are:

• Smaller vocabularies
• No unknown token

This is totally expected behavior. The byte-level BPE converts all the Unicode code points into multiple byte-level characters:

1. Each Unicode code point is decomposed into bytes (1 byte for ASCII characters, and up to 4 bytes for UTF-8 Unicode code points)
2. Each byte value gets a "visible" character assigned to it from the beginning of the Unicode table. This is especially important because there are a lot of control characters, so we can't just have a simple mapping ASCII Table character <-> byte value. So some characters get other representations, like for example the white space U+0020 becomes Ġ.

The purpose is, by doing so, you end up with an initial alphabet of 256 tokens. These 256 tokens can then be merged together to represent any other token in the vocabulary. This results in smaller vocabularies, that won't ever need an "unknown" token.

Originally posted by @n1t0 in #203 (comment)

@n1t0
Thank you for your previous responses. I have been working with a large tokenizer of a LLM, and I've noticed that the vocabulary contains a significant amount of information that like these unreadable codes.

I wonder if there are any methods or tools available to help me read and interpret the information in the tokenizer's vocabulary. For example, is there a way to map these tokens back to their original words or phrases, or any other approach to make the vocabulary more interpretable?