Can we have an example of pure C ++?

[cgisky1980]

Can we have an example of pure C ++ ?
python & pytorch is too big for the end users.

[saharNooby]

python & pytorch is too big for the end users.

I agree, but pure C++ implementation is blocked by implementing tokenizer, specifically -- Unicode normalization and Unicode regexes in 20B_tokenizer. Although RWKV World models are easier to support, since their tokenizer does not require Unicode libraries.

I myself have no plans of implementing the tokenizer in C, but I welcome PRs.

[cgisky1980]

https://github.com/koute/rwkv_tokenizer rust

[cgisky1980]

python & pytorch is too big for the end users.

I agree, but pure C++ implementation is blocked by implementing tokenizer, specifically -- Unicode normalization and Unicode regexes in 20B_tokenizer. Although RWKV World models are easier to support, since their tokenizer does not require Unicode libraries.

I myself have no plans of implementing the tokenizer in C, but I welcome PRs.

https://github.com/mlc-ai/tokenizers-cpp

[saharNooby]

@cgisky1980 Unfortunately, tokenizers-cpp will not work here:

You also need to turn on c++17 support