Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RoCBert support for Bettertransformer #542

Merged
merged 11 commits into from
Jan 12, 2023
Merged

Add RoCBert support for Bettertransformer #542

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f7aad13
Add RoCBert to overview doc
shogohida Dec 3, 2022
4cd8034
Add RoCBertLayerBetterTransformer
shogohida Dec 3, 2022
558ea42
Add RoCBertLayerBetterTransformer to init
shogohida Dec 3, 2022
5d3fa65
Add tiny-random-RoCBertModel
shogohida Dec 3, 2022
42140ff
Run make style
shogohida Dec 3, 2022
92f024f
Delete RoCBertLayerBetterTransformer
shogohida Dec 5, 2022
6c09f51
Add rocbert to init
shogohida Jan 8, 2023
61ff473
Fix test
shogohida Jan 8, 2023
228aa15
Fix test
shogohida Jan 8, 2023
97d9538
Change rocbert to roc_bert
shogohida Jan 10, 2023
4c7925e
Remove tiny-random-RoCBertModel
shogohida Jan 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/bettertransformer/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ The list of supported model below:
- [M2M100](https://arxiv.org/abs/2010.11125)
- [RemBERT](https://arxiv.org/abs/2010.12821)
- [RoBERTa](https://arxiv.org/abs/1907.11692)
- [RoCBert](https://aclanthology.org/2022.acl-long.65.pdf)
- [Splinter](https://arxiv.org/abs/2101.00438)
- [Tapas](https://arxiv.org/abs/2211.06550)
- [ViLT](https://arxiv.org/abs/2102.03334)
Expand Down
1 change: 1 addition & 0 deletions optimum/bettertransformer/models/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class BetterTransformerManager:
"mbart": ("MBartEncoderLayer", MBartEncoderLayerBetterTransformer),
"rembert": ("RemBertLayer", BertLayerBetterTransformer),
"roberta": ("RobertaLayer", BertLayerBetterTransformer),
"roc_bert": ("RoCBertLayer", BertLayerBetterTransformer),
"splinter": ("SplinterLayer", BertLayerBetterTransformer),
"tapas": ("TapasLayer", BertLayerBetterTransformer),
"vilt": ("ViltLayer", ViltLayerBetterTransformer),
Expand Down
9 changes: 9 additions & 0 deletions tests/bettertransformer/test_bettertransformer_encoder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
"hf-internal-testing/tiny-random-MarkupLMModel",
"hf-internal-testing/tiny-random-rembert",
"hf-internal-testing/tiny-random-RobertaModel",
"hf-internal-testing/tiny-random-RoCBertModel",
shogohida marked this conversation as resolved.
Show resolved Hide resolved
"hf-internal-testing/tiny-random-SplinterModel",
"hf-internal-testing/tiny-random-TapasModel",
"hf-internal-testing/tiny-xlm-roberta",
Expand Down Expand Up @@ -255,6 +256,14 @@ def test_accelerate_compatibility_single_gpu_without_keeping(self):
self.check_accelerate_compatibility_cpu_gpu(keep_original_model=False, max_memory=max_memory)


class BetterTransformersRoCBertTest(BetterTransformersEncoderTest):
all_models_to_test = ["hf-internal-testing/tiny-random-RoCBertModel"]

# unrelated issue with torch.amp.autocast with rocbert (expected scalar type BFloat16 but found Float)
def test_raise_autocast(self):
pass


class BetterTransformersEncoderDecoderTest(BetterTransformersTestMixin, unittest.TestCase):
r"""
Full testing suite of the `BetterTransformers` integration into Hugging Face
Expand Down