Post-training Activation Pruning algorithm

[yujiepan-work]

Changes

Add torch backend implementation for Post-training Activation Pruning algorithm.

The current interface is

from nncf import IgnoredScope
from nncf.experimental.torch.sparsify_activations import TargetScope

nncf.experimental.torch.sparsify_activations.sparsify_activations(
  torch_model, calibration_dataset, 
  target_sparsity_by_scope={
    TargetScope(patterns=[".*linear.*"]): 0.5,
  },
  ignored_scope=IgnoredScope(),
)

See model accuracy check at #2683 (comment)

Tests

Added unit tests at tests/torch/experimental/sparsify_activations
Added conformance tests at tests/post_training/experimental/sparsify_activations

[alexsu52]

Thanks for the contribution!

Could you add to the description any results by accuracy and performance?

[yujiepan-work]

Thanks for the contribution!

Could you add to the description any results by accuracy and performance?

Thanks for your reply! We will add this by the beginning of ww24.

[nikita-savelyevv]

Great work with the implementation, Yujie! Also, the tests are very thorough. Mostly minor comments from my side.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 187 lines in your changes missing coverage. Please review.

Project coverage is 62.10%. Comparing base (3d11e8a) to head (3c9a7b7).
Report is 49 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #2683       +/-   ##
============================================
+ Coverage    47.13%   62.10%   +14.96%     
============================================
  Files          495      486        -9     
  Lines        45986    46630      +644     
============================================
+ Hits         21675    28959     +7284     
+ Misses       24311    17671     -6640     

Files	Coverage Δ
...xperimental/torch/sparsify_activations/__init__.py	0.00% <0.00%> (ø)
.../sparsify_activations/sparsify_activations_impl.py	0.00% <0.00%> (ø)
...mental/torch/sparsify_activations/torch_backend.py	0.00% <0.00%> (ø)

... and 223 files with indirect coverage changes

Flag	Coverage Δ
COMMON	41.93% <ø> (-1.64%)	⬇️
ONNX	34.04% <0.00%> (-0.71%)	⬇️
OPENVINO	40.80% <0.00%> (+0.87%)	⬆️
TENSORFLOW	29.27% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	87.56% <ø> (+17.89%)	⬆️
torch	32.82% <0.00%> (-0.37%)	⬇️
tensorflow	93.26% <ø> (+93.26%)	⬆️
onnx	93.06% <ø> (+<0.01%)	⬆️
openvino	94.51% <ø> (+0.36%)	⬆️
ptq	82.04% <ø> (+2.64%)	⬆️

[yujiepan-work]

Model accuracy check:

Llama-2-7b-hf

	model-level sparsity	avg. zero-shot accuracy (8 tasks)	wikitext perplexity
FP16 baseline	-	71.15	8.79
FP16 sparse	25% (up/gate32%+down52%)	70.73 (-0.59%)	9.01
INT8 sparse	25% (up/gate32%+down52%)	70.88 (-0.38%)	9.01

Mixtral-8x7B-Instruct-v0.1

	model-level sparsity	avg. zero-shot accuracy (8 tasks)	wikitext perplexity
FP16 baseline	-	80.08	6.22
FP16 sparse	40% (up/gate42%+down52%)	79.70 (-0.48%)	6.56

Tasks: 'arc_easy', 'arc_challenge', 'boolq', "piqa", 'lambada_openai', 'winogrande', 'sciq', 'hellaswag'

[yujiepan-work]

Remaining issues at this moment:

• README markdown at the algo folder
• Documentation: example of using the interface
• target_sparsity_by_scope interface design

[nikita-savelyevv]

Another round of review. Awaiting addition of activation sparsity test pipeline to our CI to check the conformance test results there.

[nikita-savelyevv]

@daniil-lyakhov could you please review the PyTorch backend part of the implementation, specifically PTSparsifyActivationsAlgoBackend class?

[yujiepan-work]

@daniil-lyakhov Hi Daniil, thanks for reviewing this PR. Are there any openings that we need to change?

[daniil-lyakhov]

LGTM 👍

[nikita-savelyevv]

Conformance test build (id=12) has passed. The time it takes is about half an hour.

The only thing left is some documentation of the method.

[nikita-savelyevv]

Looks good! I can suggest only some minor tweaks

[yujiepan-work]

There is a failed test but seems not caused by this PR:

FAILED tests/common/graph/test_dot_file_rw.py::test_colons_are_replaced_in_written_dot_file - AssertionError: assert False
 +  where False = <function cmp at 0x7f52758f80d0>(PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_colons_are_replaced_in_wr0/graph.dot'), PosixPath('/home/runner/work/nncf/nncf/tests/common/data/reference_graphs/dot_rw_reference.dot'))
 +    where <function cmp at 0x7f52758f80d0> = filecmp.cmp

Update: the failed test can pass after retry

[yujiepan-work]

Thank you all for the reviews! Since we have resolved all the issues, I wonder whether this PR is ready to be merged. If there are any further changes needed, we would be glad to deal with them. 🙂

[nikita-savelyevv]

@alexsu52, please take a look. The PR should be ready for merging.

cc @AlexKoff88 @MaximProshin

[alexsu52]

Thanks for the contribution!