[DOC]: Added INT4 weight compression description #20812
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
AlexKoff88
commented
Nov 1, 2023
•
AlexKoff88
commented
- Added docs for INT4
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
|
LGTM |
ngaloppo
reviewed
Nov 1, 2023
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
|
|
||
| OpenVINO also supports models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized | ||
| with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. There is no need to do an extra step of model optimization in this case because | ||
| model conversion will ensure that int4 optimization results are preserved and model inference will benefit from it. |
There was a problem hiding this comment.
How does this work? By loading the model with OVModelForCausalLM and everything else happens automagically? Is that documented somewhere, and if so perhaps a link to there from here?
There was a problem hiding this comment.
This works as it is described, i.e. with OVModelForCausalLM . Happy to see a proposal if it is not clear here.
docs/optimization_guide/nncf/code/weight_compression_openvino.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Nico Galoppo <[email protected]>
Co-authored-by: Nico Galoppo <[email protected]>
Co-authored-by: Nico Galoppo <[email protected]>
tsavina
reviewed
Nov 2, 2023
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md
Outdated
Show resolved
Hide resolved
ngaloppo
reviewed
Nov 2, 2023
Comment on lines
54
to
56
| OpenVINO also supports models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized | ||
| with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. There is no need to do an extra step of model optimization in this case because | ||
| model conversion will ensure that int4 optimization results are preserved and model inference will benefit from it. |
There was a problem hiding this comment.
Suggested change
| OpenVINO also supports models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized | |
| with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. There is no need to do an extra step of model optimization in this case because | |
| model conversion will ensure that int4 optimization results are preserved and model inference will benefit from it. | |
| OpenVINO also supports models from Hugging Face `Transformers <https://github.com/huggingface/transformers>`__ library optimized | |
| with `GPTQ <https://github.com/PanQiWei/AutoGPTQ>`__. Those models can be loaded and converted directly with the `from_pretrained()` methods of the `Optimum Intel <https://huggingface.co/docs/optimum/main/en/intel/inference>`__ wrappers for Hugging Face models. Model conversion will ensure that int4 optimization results are preserved and model inference will benefit from it. |
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
…ight_compression.md Co-authored-by: Tatiana Savina <[email protected]>
…ight_compression.md Co-authored-by: Tatiana Savina <[email protected]>
…ight_compression.md Co-authored-by: Tatiana Savina <[email protected]>
…ight_compression.md Co-authored-by: Tatiana Savina <[email protected]>
…ight_compression.md Co-authored-by: Tatiana Savina <[email protected]>
|
I think we should proceed with the merge. @yury-gorbachev, please vote if you agree. |
tsavina
approved these changes
Nov 7, 2023
|
@tsavina, we need to have this in the release branch as well. |
allnes
pushed a commit
to allnes/openvino
that referenced
this pull request
Nov 23, 2023
* Added INT4 information into weight compression doc * Added GPTQ info. Fixed comments * Fixed list * Fixed issues. Updated Gen.AI doc * Applied comments * Added additional infor about GPTQ support * Fixed typos * Update docs/articles_en/openvino_workflow/gen_ai.md Co-authored-by: Nico Galoppo <[email protected]> * Update docs/articles_en/openvino_workflow/gen_ai.md Co-authored-by: Nico Galoppo <[email protected]> * Update docs/optimization_guide/nncf/code/weight_compression_openvino.py Co-authored-by: Nico Galoppo <[email protected]> * Applied changes * Update docs/articles_en/openvino_workflow/gen_ai.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/gen_ai.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/gen_ai.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md Co-authored-by: Tatiana Savina <[email protected]> * Update docs/articles_en/openvino_workflow/model_optimization_guide/weight_compression.md Co-authored-by: Tatiana Savina <[email protected]> * Added table with results * One more comment --------- Co-authored-by: Nico Galoppo <[email protected]> Co-authored-by: Tatiana Savina <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.