Code

Port Support Skip Special Tokens#339

Code

Closed

apaniukov wants to merge 1 commit intoopenvinotoolkit:releases/2024/6openvinotoolkit/openvino_tokenizers:releases/2024/6from apaniukov:support-skip-special-tokensapaniukov/openvino_tokenizers:support-skip-special-tokensCopy head branch name to clipboard

+43-12

Commits

Commits on Dec 6, 2024

• make skip_tokens an input for VocabDecode (parametrize detokenization/decoding) (openvinotoolkit#325)

  Show description for a7367f1
  

  pavel-esir

  authored and

  apaniukov

  committed
  a7367f1

  Copy full SHA for a7367f1