Support for Golang now or support a cli for other languages?

[xuxiaoxia96]

Hey team,
Thanks for this great library, this helped us to avoid installing the whole transformers library to be able to use the tokenizer!
Any plan for Golang binding over the Rust implementation or from scratch?

Also, where one would start to write it from scratch in Golang?

[Narsil]

I think other projects are maintaining their own: https://pkg.go.dev/github.com/gomlx/tokenizers#section-readme

We are currently not going to support due to the low amount of demand (compared to Python)

[janpfeifer]

Sorry, the github.com/gomlx/tokenizers is still under construction (and I updated the README.md to reflect that).

I stopped moving forward with it recently because for now I'm working on porting Gemma model to GoMLX, for which I'm now using the recent https://github.com/eliben/go-sentencepiece , maybe it can help you @xuxiaoxia96 ?