-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
StaticLLMPipeline: Update config #969
StaticLLMPipeline: Update config #969
Conversation
…genai into at/update-static-llm-pipeline-config
….com/TolyaTalamanov/openvino.genai into at/update-static-llm-pipeline-config
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay-ish for now but, you now, this never ends
src/cpp/src/llm_pipeline_static.cpp
Outdated
ov::AnyMap config = { | ||
{ "NPU_USE_NPUW", "YES" }, | ||
{ "NPU_COMPILATION_MODE_PARAMS", "compute-layers-with-higher-precision=Sqrt,Power,ReduceMean,Add_RMSNorm" }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So do you remember this Add vs Add_RMSNorm problem?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be just Add
perhaps
src/cpp/src/llm_pipeline_static.cpp
Outdated
const std::optional<NPUDesc>& desc) { | ||
auto config = get_baseline_common_config(); | ||
if (desc.has_value() && desc->support_max_mem_alloc_size) { | ||
config.emplace("NPUW_PMM", "NO"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Disabled PMM can be in the base (common) config I believe
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DOne
src/cpp/src/llm_pipeline_static.cpp
Outdated
ov::AnyMap get_default_common_config(const std::shared_ptr<ov::Model>& model, | ||
const std::optional<NPUDesc>& desc) { | ||
auto config = get_baseline_common_config(); | ||
if (desc.has_value() && desc->support_max_mem_alloc_size) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, is this check enough? shouldn't you check for the actual value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part is removed, will be added in next PRs
src/cpp/src/llm_pipeline_static.cpp
Outdated
auto config = get_baseline_common_config(); | ||
if (desc.has_value() && desc->support_max_mem_alloc_size) { | ||
config.emplace("NPUW_PMM", "NO"); | ||
config.emplace("NPUW_FUNCALL_FOR_ALL", "YES"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once FCFA is on, the PARALLEL_COMPILE can be ON for both models
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will take it into account for the next PR that enables FCFA, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So for it and merge., Tomorrow there will be another one. :D
This reverts commit 9c0fb7b.
No description provided.