Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow overriding special token flags in encode and decode methods #1855

Merged
merged 1 commit into from
Aug 2, 2022

Conversation

siddvenk
Copy link
Contributor

@siddvenk siddvenk commented Aug 2, 2022

Description

The tokenizer python and rust apis allow users to pass boolean flags to encode and decode methods to override special token behavior set during initialization.

This change adds apis for the user to override the behavior in java similar to python and rust.

I also updated the naming of this flag for the decoding method to match the tokenizers naming.

@siddvenk siddvenk force-pushed the tokenizer-special-tokens branch from fa8dc58 to 28a33be Compare August 2, 2022 17:10
@codecov-commenter
Copy link

codecov-commenter commented Aug 2, 2022

Codecov Report

Merging #1855 (28a33be) into master (bb5073f) will decrease coverage by 2.31%.
The diff coverage is 64.61%.

@@             Coverage Diff              @@
##             master    #1855      +/-   ##
============================================
- Coverage     72.08%   69.77%   -2.32%     
- Complexity     5126     5544     +418     
============================================
  Files           473      527      +54     
  Lines         21970    24468    +2498     
  Branches       2351     2666     +315     
============================================
+ Hits          15838    17072    +1234     
- Misses         4925     6086    +1161     
- Partials       1207     1310     +103     
Impacted Files Coverage Δ
api/src/main/java/ai/djl/modality/cv/Image.java 69.23% <ø> (-4.11%) ⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java 76.00% <ø> (ø)
...rc/main/java/ai/djl/modality/cv/output/Joints.java 71.42% <ø> (ø)
.../main/java/ai/djl/modality/cv/output/Landmark.java 100.00% <ø> (ø)
...main/java/ai/djl/modality/cv/output/Rectangle.java 72.41% <0.00%> (ø)
...i/djl/modality/cv/translator/BigGANTranslator.java 21.42% <0.00%> (-5.24%) ⬇️
...odality/cv/translator/BigGANTranslatorFactory.java 33.33% <0.00%> (+8.33%) ⬆️
...nslator/InstanceSegmentationTranslatorFactory.java 14.28% <0.00%> (-3.90%) ⬇️
.../cv/translator/SemanticSegmentationTranslator.java 0.00% <0.00%> (ø)
.../cv/translator/StyleTransferTranslatorFactory.java 40.00% <ø> (ø)
... and 424 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95d9519...28a33be. Read the comment docs.

@siddvenk siddvenk merged commit 927187b into deepjavalibrary:master Aug 2, 2022
@siddvenk siddvenk deleted the tokenizer-special-tokens branch August 2, 2022 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants