[Tutorial] Deploy Quantized Model on CUDA #4667
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anijain2305
approved these changes
Jan 9, 2020
There was a problem hiding this comment.
LGTM with some minor comments.
I was expecting to see some accuracy and performance characteristics, but I also realize that might be out of scope of this user tutorial.
Later, it might also be useful to add a developer tutorial to help TVM developers add new operators for quantization.
| ================================ | ||
| **Author**: `Wuwei Lin <https://github.com/vinx13>`_ | ||
|
|
||
| This article is an introductory tutorial of automatic quantization with TVM. |
There was a problem hiding this comment.
Maybe add a link to this discuss forum - https://discuss.tvm.ai/t/quantization-story/3920 to give a high-level idea of whats automatic quantization.
|
|
||
| ############################################################################### | ||
| # The calibration dataset should be a iterable object. We define the | ||
| # calibration dataset as a generator object in Python. In this tutorials, we |
| # When the scales are not power of two, fixed point multiplications will | ||
| # be used. | ||
| # | ||
| # For outputs, we can find the scales with data-aware quantization. |
There was a problem hiding this comment.
outputs --> intermediate feature maps
|
|
||
| import tvm | ||
| from tvm import relay | ||
| from tvm.relay import quantize as qtz |
masahi
reviewed
Jan 9, 2020
| ############################################################################### | ||
| # Import the model | ||
| # ---------------- | ||
| # We use the Relay MxNet frontent to import a model from the Gluon model zoo. |
masahi
reviewed
Jan 9, 2020
masahi
approved these changes
Jan 10, 2020
tmoreau89
reviewed
Jan 10, 2020
| **Author**: `Wuwei Lin <https://github.com/vinx13>`_ | ||
|
|
||
| This article is an introductory tutorial of automatic quantization with TVM. | ||
| Automatic quantization is one of the quantization mode in TVM. More details of the quantization story in TVM can be found `here <https://discuss.tvm.ai/t/quantization-story/3920>`_. |
There was a problem hiding this comment.
mode -> modes
details of -> details on
| # Prepare the Dataset | ||
| # ------------------- | ||
| # We will demonstrate how to prepare the calibration dataset for quantization. | ||
| # We first download the validate set of ImageNet and pre-process the dataset. |
|
|
||
|
|
||
| ############################################################################### | ||
| # The calibration dataset should be a iterable object. We define the |
| # intermediate feature maps are power of two, we can leverage bit shifting for | ||
| # multiplications. This make it computationally more efficient. In `max` mode, | ||
| # the maximum is used as the scale. Without rounding, `max` mode might have | ||
| # better accuracy in some cases. When the scales are not power of two, fixed |
tmoreau89
requested changes
Jan 10, 2020
|
Thanks @vinx13 @anijain2305 @tmoreau89 |
alexwong
pushed a commit
to alexwong/tvm
that referenced
this pull request
Feb 26, 2020
* [Tutorial] Deploy Quantized Model on CUDA * update * update * address comments
alexwong
pushed a commit
to alexwong/tvm
that referenced
this pull request
Feb 28, 2020
* [Tutorial] Deploy Quantized Model on CUDA * update * update * address comments
zhiics
pushed a commit
to neo-ai/tvm
that referenced
this pull request
Mar 2, 2020
* [Tutorial] Deploy Quantized Model on CUDA * update * update * address comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This tutorial demonstrates how to import a model using Relay frontend, run quantization and calibration passes, and perform quantized inference.
ref #4435
cc @tqchen @masahi @anijain2305 @ZihengJiang @tmoreau89