Skip to content

Releases: bnosac/tokenizers.bpe

CRAN Release 0.1.3

16 Sep 05:43
@jwijffels jwijffels
0.1.3
72ecec4
Compare
Choose a tag to compare
CRAN Release 0.1.3 Latest
Latest

CHANGES IN tokenizers.bpe VERSION 0.1.3

  • Update parallel-hashmap to 1.3.11 which fixes C++17 deprecations about rebind, construct, destroy
Assets 2
Loading

CRAN Release 0.1.2

15 Sep 07:45
@jwijffels jwijffels
0.1.2
288ff70
Compare
Choose a tag to compare
CRAN Release 0.1.2

CHANGES IN tokenizers.bpe VERSION 0.1.2

  • Drop C++11 specification in Makevars
Assets 2
Loading

CRAN Release 0.1.1

06 Jan 10:38
@jwijffels jwijffels
0.1.1
b0bc0bb
Compare
Choose a tag to compare
CRAN Release 0.1.1

CHANGES IN tokenizers.bpe VERSION 0.1.1

  • replace move with std::move to fix R CMD check warning on recent versions of clang compilers
Assets 2
Loading

CRAN Release 0.1.0

05 Aug 11:26
@jwijffels jwijffels
0.1.0
7ad4a49
Compare
Choose a tag to compare
CRAN Release 0.1.0

  • Build a BPE model

  • Tokenize using the BPE model / Reverse the tokenisation (bpe_decode / bpe_encode)

  • Load a model from disk

  • Based on YouTokenToMe commit 16fcf1d8e208f9bbbc9f522e83e6ff81708dba73
    Replaced third_party/flat_hash_map.h with https://github.com/greg7mdp/parallel-hashmap commit c4abc895b4e51cd5e0a96656bfb925edeae5ecbf

Assets 2
Loading