Add Support for DeBERTaV2

[sam-h-bean]

I would like to use DeBERTaV2 for sequence classification as a quantized model. Please let me know what needs to be done to open a PR to add this support!

[JingyaHuang]

Hi @sam-h-bean,

Before quantizing a DeBERTaV2 model, you need to be able to export it to the ONNX format, which is not supported yet. To do so, you could implement a custom OnnxConfig in transformers and add "sequence-classification" feature(if it is the only one that you are looking for) with following steps.

Feel free to reach out if you have any other question!

[sam-h-bean]

@JingyaHuang I opened a PR here! Excited to get to the quantizing once this makes its way into main. Or is there a way I can begin that work using my local copy of transformers?

[JingyaHuang]

@sam-h-bean Sure, feel free to test ORTQuantizer with your copy of transformers, I can also add the support of DeBERTa-V2 for ORTOptimizer in case you want to do some graph optimizations.

[JingyaHuang]

Closed as completed.

P.S. As DeBERTa tokenizers output token_type_ids by default and ORT doesn't take invalid inputs, token_type_ids should be removed from inputs before inference.

tokenizer = {processor_class}.from_pretrained("{checkpoint}")
model = {model_class}.from_pretrained("{checkpoint}")
-inputs = tokenizer("Optimum is nice.", return_tensors="pt")
+inputs = tokenizer("Optimum is nice.", return_tensors="pt", return_token_type_ids=False)
outputs = model(**inputs)

And for pipeline API, we are working on taking these edge cases into consideration.

[caffeinetoomuch]

I am getting Exception: Incomplete symbolic shape inference when optimizing DeBERTaV2 with fp16(AutoOptimizationConfig.O4(for_gpu=True)). Do I need to disable specific fusion?

[fxmarty]

@ice-americano Hi, could you open a new issue to track down this problem? Thanks a lot!