Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pytorch] Implement padding to bert tokenizer #1328

Merged
merged 1 commit into from
Oct 28, 2021

Conversation

frankfliu
Copy link
Contributor

Change-Id: I033baaaa1753570e62e11d2096c6ef03cd1aa48f

Description

Brief description of what this PR is about

  • If this change is a backward incompatible change, why must this change be made?
  • Interesting edge cases to note here

Change-Id: I033baaaa1753570e62e11d2096c6ef03cd1aa48f
@codecov-commenter
Copy link

Codecov Report

Merging #1328 (354ce8d) into master (bb5073f) will decrease coverage by 0.48%.
The diff coverage is 55.44%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1328      +/-   ##
============================================
- Coverage     72.08%   71.60%   -0.49%     
- Complexity     5126     5142      +16     
============================================
  Files           473      478       +5     
  Lines         21970    22256     +286     
  Branches       2351     2385      +34     
============================================
+ Hits          15838    15937      +99     
- Misses         4925     5108     +183     
- Partials       1207     1211       +4     
Impacted Files Coverage Δ
...i/djl/modality/cv/translator/BigGANTranslator.java 21.42% <ø> (-5.24%) ⬇️
...odality/cv/translator/BigGANTranslatorFactory.java 33.33% <0.00%> (+8.33%) ⬆️
...nslator/InstanceSegmentationTranslatorFactory.java 14.28% <0.00%> (-3.90%) ⬇️
.../modality/cv/translator/YoloTranslatorFactory.java 8.33% <0.00%> (-1.67%) ⬇️
...i/djl/modality/cv/translator/YoloV5Translator.java 5.69% <0.00%> (ø)
...odality/cv/translator/YoloV5TranslatorFactory.java 8.33% <0.00%> (-1.67%) ⬇️
...n/java/ai/djl/modality/nlp/bert/BertTokenizer.java 92.68% <0.00%> (-2.32%) ⬇️
...modality/nlp/embedding/TrainableWordEmbedding.java 45.00% <0.00%> (-3.34%) ⬇️
...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java 54.54% <0.00%> (-12.13%) ⬇️
...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java 15.23% <0.00%> (-0.83%) ⬇️
... and 83 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21dcfd1...354ce8d. Read the comment docs.

Copy link
Contributor

@zachgk zachgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is fine, but we need to rename maxLength because that implies it is for truncating instead of padding

@frankfliu
Copy link
Contributor Author

This PR is fine, but we need to rename maxLength because that implies it is for truncating instead of padding

maxLength should be fine, it imply both padding and truncation

@frankfliu frankfliu merged commit 866161a into deepjavalibrary:master Oct 28, 2021
@frankfliu frankfliu deleted the bert branch October 28, 2021 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants