v1.2.0

belladoreai released this 24 Mar 18:17

· 13 commits to master since this release

bd2be96

Add TypeScript types definition file
Refactor tokenizer into a Class
Allow passing custom vocab and merge data to tokenizer
Allow passing custom tests to tokenizer test runner

Assets 2