[Proposal] demo compressor #1402
Conversation
docs/en_US/Compressor/Overview.md
Outdated
| ## algorithm | ||
| We now provide some naive compression algorithm and four popular compress agorithms for users, including two pruning algorithm and two quantization algorithm. | ||
| Below is a list of model compression algorithms supported in our compressor | ||
| | Name | Paper | |
There was a problem hiding this comment.
this table cannot be correctly rendered
docs/en_US/Compressor/Overview.md
Outdated
| We use the instrumentation method to insert a node or function after the corresponding position in the model. | ||
| <br> | ||
| When compression algorithm designer implements one prune algorithm, he only need to pay attention to the generation method of mask, without caring about applying the mask to the garph. | ||
| ## algorithm |
docs/en_US/Compressor/Overview.md
Outdated
|
|
||
| Tensorflow code | ||
| ``` | ||
| nni.compressors.tfCompressor.LevelPruner(sparsity=0.8).compress(model_graph) |
There was a problem hiding this comment.
format (camel case) of package names 'tfCompressor' and 'torchCompressor' are inconsistent with current nni package names.
| r_qw = 2*qw - 1 | ||
| return r_qw | ||
|
|
||
| except ImportError: |
setup.py
Outdated
| @@ -58,7 +58,8 @@ def read(fname): | |||
| 'schema', | |||
| 'PythonWebHDFS', | |||
| 'colorama', | |||
| 'sklearn' | |||
| 'sklearn', | |||
| 'pyyaml' | |||
There was a problem hiding this comment.
We have already required ruamel.yaml.
Should not add another YAML library.
| @@ -0,0 +1,125 @@ | |||
| from nni.compressors.tf_compressor import AGPruner | |||
There was a problem hiding this comment.
Examples should be placed in examples folder, not sdk.
src/sdk/pynni/requirements.txt
Outdated
| @@ -8,3 +8,6 @@ hyperopt | |||
|
|
|||
| # metis tuner | |||
| sklearn | |||
|
|
|||
| # compresser | |||
| pyyaml | |||
src/sdk/pynni/setup.py
Outdated
| @@ -36,7 +36,8 @@ def read(fname): | |||
| 'json_tricks', | |||
| 'numpy', | |||
| 'scipy', | |||
| 'coverage' | |||
| 'coverage', | |||
| 'pyyaml' | |||
src/webui/yarn.lock
Outdated
| @@ -3,73 +3,72 @@ | |||
|
|
|||
There was a problem hiding this comment.
Checkout this file from master
setup.py
Outdated
| @@ -27,7 +27,7 @@ def read(fname): | |||
|
|
|||
| setup( | |||
| name = 'nni', | |||
| version = '999.0.0-developing', | |||
| version = 'v0.8-347-g0538d30', | |||
| pass | ||
|
|
||
|
|
||
| class TfLayerInfo: |
There was a problem hiding this comment.
add docstring for this class
| return configure_file | ||
|
|
||
| class TfPruner(TfCompressor): | ||
| """TODO""" |
| whiltlist = ['Conv2D', 'DepthwiseConv2dNative'] | ||
| return [ TfLayerInfo(op) for op in model.get_operations() if op.type in whiltlist ] | ||
|
|
||
| def _tf_default_get_configure(configure_list, layer_info): |
There was a problem hiding this comment.
add docstring for functions
| elif layer_info.name in config.get('support_op', []): | ||
| configure = config | ||
|
|
||
| return configure |
There was a problem hiding this comment.
should properly check whether configure is {} when calling this function
| return configure | ||
|
|
||
| def _tf_default_load_configure_file(config_path, class_name): | ||
| print('load CLASS:{0} from PATH:{1}'.format(class_name, config_path)) |
There was a problem hiding this comment.
please using log rather than print
| self._instrument_layer(layer_info) | ||
|
|
||
| def _instrument_layer(self, layer_info): | ||
| # it seems the graph editor can only swap edges of nodes or remove all edges from a node |
| new_weight = weight * mask | ||
| tf.contrib.graph_editor.swap_outputs(weight_op, new_weight.op) | ||
|
|
||
| def update_epoch(self, epoch, sess): |
| def update_epoch(self, epoch, sess): | ||
| pass | ||
|
|
||
| def step(self, sess): |
There was a problem hiding this comment.
add docstring. And why update_epoch and step are in TfPruner rather than TfCompressor?
| """ | ||
| super().__init__() | ||
| self.configure_list = [] | ||
| if isinstance(configure_list, list): |
There was a problem hiding this comment.
do you accept config file path?
| else: | ||
| raise ValueError('please init with configure list') | ||
|
|
||
| def load_configure(self, config_path): |
There was a problem hiding this comment.
is this function called by others?
| final_sparsity | ||
| start_epoch | ||
| end_epoch | ||
| frequency |
There was a problem hiding this comment.
please give more description. what is the meaning of frequency?
|
|
||
| class AGPruner(TfPruner): | ||
| def __init__(self, configure_list): | ||
| """ |
There was a problem hiding this comment.
brief describe how this algorithm works
| final_sparsity = configure.get('final_sparsity', 0) | ||
| initial_sparsity = configure.get('initial_sparsity', 0) | ||
|
|
||
| if end_epoch <= start_epoch or initial_sparsity >= final_sparsity: |
There was a problem hiding this comment.
is it an error if end_epoch < start_epoch?
| target_sparsity = self.compute_target_sparsity(layer_info) | ||
| threshold = tf.contrib.distributions.percentile(weight, target_sparsity * 100) | ||
| mask = tf.stop_gradient(tf.cast(tf.math.greater(weight, threshold), weight.dtype)) | ||
| print('tensor weight', weight) |
There was a problem hiding this comment.
use log debug here if you think logging weight is helpful during debug
|
|
||
| target_sparsity = self.compute_target_sparsity(layer_info) | ||
| threshold = tf.contrib.distributions.percentile(weight, target_sparsity * 100) | ||
| mask = tf.stop_gradient(tf.cast(tf.math.greater(weight, threshold), weight.dtype)) |
There was a problem hiding this comment.
please explain why use tf.stop_gradient here in comments
setup.py
Outdated
| @@ -27,7 +27,7 @@ def read(fname): | |||
|
|
|||
| setup( | |||
| name = 'nni', | |||
| version = '999.0.0-developing', | |||
| version = 'v0.8-263-g1f92408', | |||
There was a problem hiding this comment.
this is a temporary change by NNI installation script. Please remove this commit.
| @@ -0,0 +1,115 @@ | |||
| from nni.compressors.tfCompressor import AGPruner | |||
There was a problem hiding this comment.
where's nni.get_next_parameter() and nni.report_final_results()?
| @@ -0,0 +1,86 @@ | |||
| try: | |||
There was a problem hiding this comment.
why catch ImportError here? Also the way you handle it is problematic, which will cause other import errors.
| @@ -0,0 +1,54 @@ | |||
| try: | |||
src/webui/yarn.lock
Outdated
| @@ -3,73 +3,72 @@ | |||
|
|
|||
|
|
|||
| "@ant-design/icons-react@~1.1.2": | |||
There was a problem hiding this comment.
This file should not be changed.
* [Proposal] demo compressor (#1402) model compression * update doc for model compression (#1509) * Update Overview.md * Change Doc (#1510) * refactor compression sdk (#1562) * refactor compression sdk * bugfix * bugfix * update ut * Sync model compression doc and implementation (#1575) * update doc * formatting * bugfix * add import to examples
now we provide compressors for pytorch and Tensorflow, including 4 compress algorithms:
Sensitivity Pruner
Learning both Weights and Connections for Efficient Neural Networks
https://arxiv.org/pdf/1506.02626.pdf
Auto Gradual Pruner
To prune, or not to prune: exploring the efficacy of pruning for model compression
https://arxiv.org/pdf/1710.01878.pdf
Quantize Aware Training Quantizer
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
https://arxiv.org/pdf/1712.05877.pdf
DoReFa Quantizer
DOREFA-NET: TRAINING LOW BITWIDTH CONVOLUTIONAL NEURAL NETWORKS WITH LOW BITWIDTH GRADIENTS
https://arxiv.org/pdf/1606.06160.pdf
For pytorch, we can just add new algorithm in it's forward method to instrument a layer
For tensorflow, we can modify graph by swapping output of new weight and old weight
To implement pruner, we use mask and update weight information
To implement quantizer, we use just replace old weight data with new one