Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition to default int4 compression configs from optimum-intel #689

Conversation

nikita-savelyevv
Copy link
Contributor

@nikita-savelyevv nikita-savelyevv commented Jul 25, 2024

Changes

  • For 4BIT_DEFAULT, the compression config is now always imported from optimum-intel
  • Remove all default configs from openvino.genai, rely only on default configs in optimum-intel
  • Adopt dataset preparation logic from optimum-intel

Breaking changes:

  • Old compression dataset specification is deprecated. For example "wikitext,wikitext-2-v1,train[:1000],text" will not work anymore. A string "wikitext2" should be used instead.
  • Compression with data-aware methods will produce different results, potentially worse.
  • Default int4 configs will no longer be matched by folder names. To match the default config, one of these options must hold:
    • model is exported with python convert.py -m <model_id>
    • model is exported with python convert.py -m <model_path>, where model_path contains a config.json file with a _name_or_path field containing a model id
    • model is exported with python convert.py -m <model_path>, where model_path contains a config.json file with a _name_or_path field containing another model_path_2, such that the condition from the bullet point above holds for it

Related tickets
147470

@github-actions github-actions bot added the category: llm_bench Label for tool/llm_bench folder label Jul 25, 2024
@nikita-savelyevv nikita-savelyevv changed the title Extend how optimum-intel configs are inherited Transition to default int4 compression configs from optimum-intel Jul 26, 2024
@nikita-savelyevv nikita-savelyevv force-pushed the inherit-default-configs-from-optimum branch 2 times, most recently from 2420c8f to 0207904 Compare August 20, 2024 11:42
@nikita-savelyevv nikita-savelyevv marked this pull request as ready for review August 20, 2024 13:04
@nikita-savelyevv nikita-savelyevv added this pull request to the merge queue Aug 26, 2024
@eaidova eaidova removed this pull request from the merge queue due to a manual request Aug 26, 2024
@eaidova eaidova merged commit d6bd822 into openvinotoolkit:master Aug 26, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: llm_bench Label for tool/llm_bench folder
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants