Skip to content

Latest commit

 

History

History
1073 lines (597 loc) · 31.9 KB

CHANGELOG.md

File metadata and controls

1073 lines (597 loc) · 31.9 KB
Raw

CHANGELOG

v0.12.0 (2024-08-21)

Feature

  • feat: add jamba 1.5 tokenizers (mini and large) (#96)

  • feat: add jamba 1.5 tokenizers (mini and large)

  • test: fix value error on conftest.py

  • refactor: rename file name

  • chore: fix path

  • fix: enter (e052b22)

v0.11.3 (2024-08-05)

Chore

  • chore(release): v0.11.3 [skip ci] (5be5b59)

  • chore(deps-dev): bump authlib from 1.2.1 to 1.3.1 (#91)

Bumps authlib from 1.2.1 to 1.3.1.

updated-dependencies:

  • dependency-name: authlib dependency-type: indirect ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (fe24518)

  • chore(deps-dev): bump black from 23.3.0 to 24.3.0 (#90)

Bumps black from 23.3.0 to 24.3.0.

updated-dependencies:

  • dependency-name: black dependency-type: direct:development ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (5257f3d)

  • chore(deps-dev): bump python-semantic-release from 8.3.0 to 8.7.0 (#80)

Bumps python-semantic-release from 8.3.0 to 8.7.0.

updated-dependencies:

  • dependency-name: python-semantic-release dependency-type: direct:development update-type: version-update:semver-minor ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Asaf Gardin <[email protected]> Co-authored-by: Asaf Joseph Gardin <[email protected]> (376894f)

Ci

  • ci: allow running tests and semantic checks on external prs (#93) (c08604d)

Fix

  • fix(deps): bump requests from 2.31.0 to 2.32.2 (#92)

Bumps requests from 2.31.0 to 2.32.2.

updated-dependencies:

  • dependency-name: requests dependency-type: indirect ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Asaf Joseph Gardin <[email protected]> (3216898)

v0.11.2 (2024-06-20)

Chore

  • chore(release): v0.11.2 [skip ci] (57e712e)

Fix

  • fix: add missing space tokens init on jursssic (#89)

  • fix: missing space tokens init on jursssic, fix import in example

  • test: add test for space_tokens init (4ec43a0)

v0.11.1 (2024-06-20)

Chore

  • chore(release): v0.11.1 [skip ci] (07bf0cd)

Fix

  • fix: Added backwards compatibility for jamba tokenizer (#88)

  • fix: Added backwards compatibility for jamba tokenizer

  • fix: Added to async

Co-authored-by: asafg <[email protected]> (9ad3475)

v0.11.0 (2024-06-18)

Chore

  • chore(release): v0.11.0 [skip ci] (0df321c)

Feature

  • feat: Async tokenizer (#86)

  • feat: support async, wip

  • feat: fix and add tests, examples, update readme

  • fix: poetry lock

  • fix: anyio -> aiofiles

  • fix: try 3.8

  • fix: remove 3.7 from tests

  • fix: poetry lock

  • fix: add 3.7 back

  • fix: poetry lock

  • fix: poetry.lock

  • ci: pipenv

  • fix: pipenv

  • fix: pipenv

  • fix: pyproject

  • fix: lock

  • fix: version

  • fix: Removed aiofiles

  • ci: update python version,

  • fix: switch from aiofiles to anyio, remove redundant comments

  • chore: poetry lock

  • fix: disable initializing async classes directly, cr comments

  • test: fix import

  • ci: add asyncio-mode to test workflow

  • fix: to_thread -> run_in_executor

  • ci: add asyncio

  • fix: cr comments

  • fix: cr comments

Co-authored-by: asafg <[email protected]> (3006cda)

v0.10.0 (2024-06-16)

Chore

  • chore(release): v0.10.0 [skip ci] (1178ba7)

Feature

  • feat: remove python 3.7 support (#87) (58482ab)

v0.9.1 (2024-05-14)

Chore

  • chore(release): v0.9.1 [skip ci] (a260d76)

Fix

  • fix: depend on less restrictive version of tokenizers (#85) (eab6a14)

v0.9.0 (2024-03-28)

Chore

  • chore(release): v0.9.0 [skip ci] (974df9e)

Feature

  • feat: Jamba instruct tokenizer (#84) (88ff9af)

v0.8.2 (2024-03-11)

Chore

  • chore(release): v0.8.2 [skip ci] (1146741)

Ci

  • ci: add Python 3.12 to test matrix (#82)

  • ci: add Python 3.12 to test matrix

  • chore: use sentencepiece 0.2.0 or higher

  • fix: update poetry.lock (8084117)

Fix

Unknown

v0.8.1 (2024-01-07)

Chore

  • chore(release): v0.8.1 [skip ci] (fcacbf8)

Fix

  • fix: re-ordered parameters in ctor to avoid a breaking change (#79) (6c1b608)

v0.8.0 (2024-01-03)

Chore

  • chore(release): v0.8.0 [skip ci] (c8b54df)

  • chore(deps-dev): bump pytest from 7.2.1 to 7.4.4 (#75)

Bumps pytest from 7.2.1 to 7.4.4.

updated-dependencies:

  • dependency-name: pytest dependency-type: direct:development update-type: version-update:semver-minor ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: asafgardin <[email protected]> (081dda3)

Feature

  • feat: Add start_of_line to decode (#77)

  • feat: Add start_of_line param to decode

  • test: added unittest with start_of_line=True and False (182a8d1)

v0.7.0 (2024-01-02)

Chore

  • chore(release): v0.7.0 [skip ci] (26f34b2)

Feature

  • feat: Init tokenizer from filehandle (#76)

  • feat: allow creating JurassicTokenizer from model file handle

  • fix: Add default for model_path and model_file_handle

  • feat: Add JurassicTokenizer.from_file_path classmethod

  • fix: remove model_path=None in JurassicTokenizer.from_file_handle

  • fix: rename _assert_exactly_one to _validate_init and make it not static

  • refactor: semantics

  • test: Added tests

Co-authored-by: Asaf Gardin <[email protected]> (dcb73a7)

v0.6.0 (2023-12-28)

Chore

  • chore(release): v0.6.0 [skip ci] (7b8348d)

  • chore: add test case for encode with is_start=False (#74)

  • chore: add test case for encode with is_start=False

  • fix: split is_start=False to a different testcase (77c0a39)

Feature

  • feat: Add decode with offsets (#73)

  • feat: Add decode_with_offsets() to JurassicTokenizer

  • refactor: remove kwargs from decode_with_offsets since it's not used

  • chore: Add unittest for decode and for offsets

  • fix: test only decode_with_offsets

  • fix: dummy for returned offsets in decode_with_offsets (a5a7bb4)

  • feat: Add the is_start parameter to JurassicTokenizer.encode() (#72)

  • feat: Add the is_start parameter to JurassicTokenizer.encode()

  • refactor: take 'is_start' from kwargs (296bda5)

v0.5.0 (2023-12-28)

Chore

  • chore(release): v0.5.0 [skip ci] (96f384f)

Feature

  • feat: Add more special tokens (#71)

  • fix: commitizen tag starts with "v"

  • feat: add eos_id

  • feat: Add newline_id

  • fix: typo "_newline_piece" instead of "newline_piece"

  • fix: newline_id already existed as "private". Just make it "public"

  • fix: forgot to rename everywhere (9a9e1a8)

Fix

  • fix: commitizen tag starts with "v" (#70) (cf495ad)

v0.4.0 (2023-12-28)

Chore

  • chore(release): v0.4.0 [skip ci] (b761edc)

Feature

  • feat: add pad_id and bos_id to jurassic_tokenizer (#69) (ffb2ce3)

v0.3.11 (2023-12-27)

Chore

  • chore(release): v0.3.11 [skip ci] (5280149)

Fix

  • fix: BaseTokenizer in init (#68) (3cc71e7)

v0.3.10 (2023-12-27)

Chore

  • chore(release): v0.3.10 [skip ci] (1601535)

  • chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (#64)

Bumps safety from 2.3.4 to 2.3.5.

updated-dependencies:

  • dependency-name: safety dependency-type: direct:development update-type: version-update:semver-patch ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (95696bb)

  • chore(deps-dev): bump ruff from 0.0.285 to 0.1.8 (#63)

Bumps ruff from 0.0.285 to 0.1.8.

updated-dependencies:

  • dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-minor ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (81123d3)

  • chore(deps-dev): bump black from 22.12.0 to 23.3.0 (#61)

Bumps black from 22.12.0 to 23.3.0.

updated-dependencies:

  • dependency-name: black dependency-type: direct:development update-type: version-update:semver-major ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (7190d28)

  • chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (#60)

Bumps safety from 2.3.4 to 2.3.5.

updated-dependencies:

  • dependency-name: safety dependency-type: direct:development update-type: version-update:semver-patch ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (2fa7bef)

Fix

  • fix: JurassicTokenizer in init (#67) (253ae07)

Refactor

  • refactor: Added all in init (#65)

  • refactor: Added all in init

  • fix: tests

  • refactor: added version to all (c0d9286)

  • refactor: sentencepiece version to support all patch versions (#66) (845008c)

v0.3.9 (2023-11-27)

Chore

  • chore(release): v0.3.9 [skip ci] (84f17da)

  • chore: add github badges (#58) (821455c)

  • chore(deps-dev): bump urllib3 from 2.0.4 to 2.0.7 (#57)

Bumps urllib3 from 2.0.4 to 2.0.7.

updated-dependencies:

  • dependency-name: urllib3 dependency-type: indirect ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (93ef6d6)

Fix

  • fix: Modify badges (#59)

  • docs: fixed url

  • fix: inline

  • fix: README.md (20b7090)

v0.3.8 (2023-11-26)

Chore

  • chore(release): v0.3.8 [skip ci] (645ff5e)

Fix

  • fix: readme example (#56) (b713da8)

v0.3.7 (2023-11-23)

Chore

  • chore(release): v0.3.7 [skip ci] (a181ae6)

Ci

  • ci: workflow dispatch for release (#54) (dbf5609)

  • ci: Automate pypi publish (#53)

  • ci: Automate pypi publish on new release

  • fix: Remove comment

  • fix: title of action

  • fix: title of action (7c04fda)

Fix

  • fix: Examples in readme (#55)

  • ci: workflow dispatch for release

  • docs: Updated readme with more examples

  • docs: Added docs to base class (94f3a3c)

v0.3.6 (2023-11-22)

Chore

  • chore(release): v0.3.6 [skip ci] (550644d)

Ci

  • ci: added python version to pypi publish (#50) (83cbbea)

  • ci: exclude changelog (#48) (9222c55)

Documentation

Fix

  • fix: support ai21_tokenizer.version (#52) (13944eb)

v0.3.5 (2023-11-22)

Chore

  • chore(release): v0.3.5 [skip ci] (217b14e)

Fix

  • fix: newline n and prettier (#47)

  • fix: newline n and prettier

  • fix: exclude

  • fix: exclude from pretty (f5c9204)

v0.3.4 (2023-11-22)

Chore

  • chore(release): v0.3.4 [skip ci] (be67c79)

Fix

v0.3.3 (2023-11-22)

Chore

  • chore(release): v0.3.3 [skip ci] (f6db520)

Ci

  • ci: on push release (#43) (95bfc95)

Documentation

Fix

v0.3.2 (2023-11-21)

Chore

  • chore(release): v0.3.2 [skip ci] (f3b2e73)

Ci

  • ci: Added newline sequence (#41) (63a1898)

Fix

  • fix: Version path (#42)

  • fix: version

  • fix: version variables

  • fix: name (ee4d744)

v0.3.1 (2023-11-21)

Chore

  • chore(release): v0.3.1 [skip ci] (2eebeb8)

Fix

  • fix: test 2 (#40) (2028613)

  • fix: Test bump 1 (#39)

  • fix: crlf forbid

  • fix: test 1 (26da29e)

v0.3.0 (2023-11-21)

Chore

  • chore(release): v0.3.0 [skip ci] (0c4ada0)

Documentation

  • docs: Release md update before publish (#36)

  • fix: Added support for both str and path

  • fix: rename package

  • fix: updated pre commits and added new one

  • docs: Updated docs

  • ci: down grade

  • docs: Added another example (18ccbeb)

  • docs: CONTRIBUTING.md (#35)

  • docs: CONTRIBUTING.md

  • ci: end_of_line fix

  • docs: inv test (e282440)

Feature

  • feat: Added char for testing (#37) (40d3feb)

Fix

  • fix: string example (#38) (833038c)

v0.2.0 (2023-11-21)

Chore

  • chore(release): v0.2.0 [skip ci] (8988faa)

  • chore(deps-dev): bump black from 22.12.0 to 23.3.0 (#32)

Bumps black from 22.12.0 to 23.3.0.

updated-dependencies:

  • dependency-name: black dependency-type: direct:development update-type: version-update:semver-major ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: asafgardin <[email protected]> (bb4986e)

Feature

  • feat: Tokenizer factory (#31)

  • feat: Added tokenizer abc and factory

  • fix: api to receive default and none

  • fix: example

  • fix: factory and tests

  • fix: rename base

  • fix: rename base class

  • fix: rename package

  • fix: example

  • fix: readme and tasks

  • docs: factory class

  • docs: renames

  • fix: directory hierarchy in tests

  • fix: rename package

  • chore(release): v0.1.2 [skip ci]

  • fix: rename package

  • ci: example

  • fix: assert in example

  • fix: src_path

Co-authored-by: github-actions <[email protected]> (e55cd1d)

Fix

v0.1.2 (2023-11-21)

Chore

  • chore(release): v0.1.2 [skip ci] (5b1dc14)

  • chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (#28)

Bumps safety from 2.3.4 to 2.3.5.

updated-dependencies:

  • dependency-name: safety dependency-type: direct:development update-type: version-update:semver-patch ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (28118ac)

  • chore(deps-dev): bump pytest-mock from 3.10.0 to 3.11.1 (#24)

Bumps pytest-mock from 3.10.0 to 3.11.1.

updated-dependencies:

  • dependency-name: pytest-mock dependency-type: direct:development update-type: version-update:semver-minor ...

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: asafgardin <[email protected]> (0a36f21)

Ci

  • ci: Remove install from publish (#29)

  • ci: Removed install dependency

  • ci: changlog changes (f8e8392)

  • ci: dependabot pr limit (#27)

  • ci: dependabot pr limit

  • ci: dependabot pr limit (9b2c4f8)

Fix

  • fix: workflow dispatch for release action (#33) (f81b4ab)

v0.1.1 (2023-11-20)

Chore

  • chore(release): v0.1.1 [skip ci] (f5150f5)

Fix

  • fix: used PAT (#26)

  • test: write to main

  • fix: token

  • test: debug

  • test: debugging tokens

  • test: uncomment

  • test: write to main

  • fix: token

  • test: debug

  • test: debugging tokens

  • test: uncomment

  • fix: Changed to main (2ff12c9)

v0.1.0 (2023-11-20)

Chore

  • chore(release): v0.1.0 [skip ci] (48607f9)

Ci

  • ci: Create dependabot.yml (#19)

  • ci: Create dependabot.yml

  • fix: commit-message prefix

  • fix: Added more config to dependabot action (23faaa8)

Feature

  • feat: Pypi publish (#18)

  • feast: Added setup.py

  • feast: Added publish.yaml (77ee751)

  • feat: test PAT (#9) (a10b6a4)

  • feat: Added semantic prs actions (#8) (afab5ff)

Fix

  • fix: Added permissions (#25)

  • fix: Added permissions

  • fix: permissions location

  • fix: verbose

  • fix: Removed bad input "root_options" (d684575)

  • fix: Change token (#17)

  • fix: token key

  • fix: token github

  • fix: token github cls

  • fix: token github cls (ec4f35b)

  • fix: Change token (#16)

  • fix: token key

  • fix: token github (d876a43)

  • fix: token key (#15) (0b76344)

  • fix: keys (#14) (b064ea6)

  • fix: Test token (#11)

  • feat: test PAT

  • feat: test github token

  • fix: PAT (94f64b6)

  • fix: Test token (#10)

  • feat: test PAT

  • feat: test github token (52484fe)

  • fix: root_options verbose (#6) (220ba5b)

  • fix: Release action test (#5)

  • fix: Added release step

  • fix: branch name for testing

  • fix: Removed comment

  • chore(release): v0.0.1 [skip ci]

  • fix: branch rename

  • fix: Removed CHANGELOG.md

Co-authored-by: github-actions <[email protected]> (fae5423)

Test

  • test: Test token (#13)

  • feat: test PAT

  • feat: test github token

  • fix: PAT

  • test: Added test step

  • test: Added test step

  • test: Added token to use (d445e36)

  • test: Test token (#12)

  • feat: test PAT

  • feat: test github token

  • fix: PAT

  • test: Added test step

  • test: Added test step (9619c3e)

Unknown

  • Add kwargs to functions (#7)

  • feat: added kwargs

  • test: Added tests (efecff9)

  • Release action (#4)

  • feat: Added release action

  • fix: Removed unnecessary code

  • fix: testing on branch

  • fix: removed node install

  • fix: Removed unnecessary step

  • fix: base-branch

  • fix: removed code

  • 0.0.2

  • feat: python-semantic-release test

  • fix: branch name

  • fix: branch name in .toml

  • fix: change from branch to match

  • fix: Added release_action part

  • chore(release): v0.1.0 [skip ci]

  • refactor: removed CHANGELOG.md

  • fix: branch to main

  • feat: Added version.py to version_variable

  • feat: Upgraded python-semantic-release

  • feat: Added python-semantic-release

  • fix: Removed unnecessary file

  • fix: Changed version

Co-authored-by: github-action <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <[email protected]> (694683a)

  • Add code (#2)

  • feat: Jurassic tokenizer

  • fix: remove is_start

  • fix: add types

  • fix: add types

  • chore: extracted utils

  • fix: simplified tokenizer even more

  • fix: simplified tokenizer even more

  • feat: Added tests

  • feat: exposed prop

Co-authored-by: Asaf Gardin <[email protected]> (6b80a05)

  • First commit (#1)

  • feat: init project (f50565e)

  • Initial commit (556d3e6)