OV Performance Hints (CPU and GPU logic for selecting the actual configs), while AUTO/MULTI are passing them thru) #6993
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
myshevts
requested review from
ilya-lavrenov,
dmitry-gorokhov,
nkogteva,
vladimir-paramuzov,
wangleis and
a team
openvino-pushbot
added
category: CPU
myshevts
force-pushed
the
perf-hints-master
branch
4 times, most recently
from
296825c to
193375b
Compare
nkogteva
reviewed
Aug 12, 2021
|
@mashoujiang fyi, this PR passes the perf hints thru the AUTO (and MULTI) |
myshevts
force-pushed
the
perf-hints-master
branch
4 times, most recently
from
ab27fff to
2f79fd4
Compare
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
Co-authored-by: Tatiana Savina <[email protected]>
wangleis
reviewed
Aug 25, 2021
wangleis
reviewed
Sep 6, 2021
wangleis
approved these changes
Sep 6, 2021
myshevts
force-pushed
the
perf-hints-master
branch
from
1a7a8e2 to
cdc0655
Compare
nkogteva
reviewed
Sep 7, 2021
nkogteva
approved these changes
Sep 8, 2021
myshevts
force-pushed
the
perf-hints-master
branch
from
df26364 to
6af6358
Compare
vladimir-paramuzov
approved these changes
Sep 10, 2021
dmitry-gorokhov
approved these changes
Sep 13, 2021
akuporos
pushed a commit
to akuporos/openvino
that referenced
this pull request
Sep 29, 2021
…igs), while AUTO/MULTI are passing them thru) (openvinotoolkit#6993) * rebasing the perf-modes-2021.3 to the 2021.4 Caveats: the (explicit) setting #streams is not disabled (as it was before for experiments with DLBenchmark), and the logic slighlty differ (streamsSet) (cherry picked from commit 1ae1edc) * overriding streams (to force the TPUT mode to the DLBenchnark) (cherry picked from commit 7f506cd) * disabling reducing #streams to fully mimic baseline c4df94d of the 2021.3 (before experiments) (cherry picked from commit 85073dd) * clang/identation (cherry picked from commit 050a415) * splitting the Transformation to general and CPU specific. Now hopefully,this fully mimics the baseline c4df94d of the 2021.3 (before experiments), as the streams reduce num (as well as early exit on GRU/LSTM/TensorIterator) is deisabled (cherry picked from commit e98b2c1) * disabling GRU/LSTM/TI + reducing of streams + 5D considered compute-limited only for int8 (cherry picked from commit 32b8d80) * refactored to avoid compute_limited_ratio, reverted the reducing #streams, removed LSTM from limitations (cherry picked from commit f2b9721) * isa-based threshold logic (cherry picked from commit b218457) * mode->hint (cherry picked from commit ec20aa8) * optional PERFORMANCE_HINT_NUM_REQUESTS (cherry picked from commit 5a3883e) * moving the perfHints to the common OV config class + initial tests (CPU only, as the actual AUTO/MULTI should be accommodated on the master) (cherry picked from commit (then fixed)45bafe7d527f466507dea0693aeed51be4ebf776) * AUTO support for PerfHints * MULTI support for PerfHints * Enabling Perf hints for the GPU plugin * brushing settings output a bit * disabling "throughput" perf hint being default (until OV 2.0) * uncommenting the logic which was disabled to force the DLBenchmark to use the throughput mode by default * removing dead and experimental code, and debug printfs * clang/code-style * code-review remarks * Moved the output of the actual params that the hint produced to the right place * aligning MULTI's GetConfig beh to HETERO's as captured in the preso (CVS-59960) ratified with the ArchForum * clang * benchmark_app brushing * Update inference-engine/samples/benchmark_app/README.md * propagating the perf hints thru one more scenario in the merged AUTO-MULTI * fixed mispint * Python benchmark_app update for perf hints * addresssing reviewers comments on the python benchmark_app * simplifying/brushing logic a bit * refactor the heuristic to the separate file (to be shared with iGPU soon) * refactor conversion of modes to the specific GPU config per feedback from Vladimir
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For the GPU (until Auto-Batching) the logic is very simple (and mimicks the benchmark_app), for the CPU in contrast the network-based heuristics selects the #streams.
@ArtemySkrebkov-intel and @rzubarev fyi