Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update compression config for openlm-research/open_llama_3b_v2 #860

Conversation

nikita-savelyevv
Copy link
Collaborator

@nikita-savelyevv nikita-savelyevv commented Aug 7, 2024

Reduce PPL for compressed openlm-research/open_llama_3b_v2 by removing all_layers=True and adding AWQ.

Precision all_layers AWQ PPL
FP16 12.40
INT4_SYM False False 13.45
INT4_SYM True False 13.73
INT4_ASYM False True 12.77
INT4_ASYM False False 13.15
INT4_ASYM True False 13.36

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@nikita-savelyevv
Copy link
Collaborator Author

@KodiaqQ please take a look

@KodiaqQ
Copy link
Contributor

KodiaqQ commented Aug 9, 2024

@nikita-savelyevv, you've provided numbers in the description with the ASYM mode, while configuration contains "sym": True. Why so?

@nikita-savelyevv
Copy link
Collaborator Author

@KodiaqQ Thanks for noticing. There were some errors in the collected metrics. I've updated the metrics in the PR description and changed the mode to asymmetric.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@AlexKoff88 AlexKoff88 merged commit 1b7bd9f into huggingface:main Aug 16, 2024
13 of 17 checks passed
IlyasMoutawwakil pushed a commit that referenced this pull request Aug 16, 2024
* Remove compression with all_layers=True for openlm-research/open_llama_3b_v2

* Fix sym parameter

* Add AWQ
github-merge-queue bot pushed a commit to openvinotoolkit/openvino.genai that referenced this pull request Aug 19, 2024
eaidova pushed a commit to openvinotoolkit/openvino.genai that referenced this pull request Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants