Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

[0.9.x] Add int8 command #1174

Merged
merged 2 commits into from
Feb 25, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions scripts/bert/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -223,22 +223,22 @@ version of `mxnet-mkl <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/inde
Sentence Classification
+++++++++++++++++++++++

+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
| Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Log | Command |
+===========+===================+===============+===============+=========+=========+=====+=========+
| MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 | | |
+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
| SST-2 | bert_12_768_12 | 93.23 | 93.00 | | | | |
+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+
| Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Command |
+===========+===================+===============+===============+=========+=========+========================================================================================================================+
| MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_MRPC_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+
| SST-2 | bert_12_768_12 | 93.23 | 93.00 | | |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_SST_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+

Question Answering
++++++++++++++++++

+-----------+-------------------+---------+---------+---------+---------+-----+---------+
| Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Log | Command |
+===========+===================+=========+=========+=========+=========+=====+=========+
| SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 | | |
+-----------+-------------------+---------+---------+---------+---------+-----+---------+
+-----------+-------------------+---------+---------+---------+---------+----------------------------------------------------------------------------------------------------------------------------+
| Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Command |
+===========+===================+=========+=========+=========+=========+============================================================================================================================+
| SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_squad1.1_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------+---------+---------+---------+----------------------------------------------------------------------------------------------------------------------------+

For all model settings above, we use a subset of evaluation dataset for calibration.

Expand Down