Merge branch 'releases/2024/3' into cmakelist-update-24-3

openvinotoolkit · Aug 5, 2024 · cefa6ab · cefa6ab
2 parents 3027ff7 + d1cfc53
commit cefa6ab
Show file tree

Hide file tree

Showing 9 changed files with 316 additions and 457 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -116,16 +116,18 @@
 # Documentation
 /docs/  @openvinotoolkit/openvino-docs-maintainers
 /docs/CMakeLists.txt @openvinotoolkit/openvino-ie-maintainers
+/**/*.rst  @openvinotoolkit/openvino-docs-maintainers
 /**/*.md  @openvinotoolkit/openvino-docs-maintainers
 /**/*.svg  @openvinotoolkit/openvino-docs-maintainers
-/docs/MO_DG/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-mo-maintainers
-/docs/OV_Runtime_UG/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
-/docs/IE_PLUGIN_DG/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
-/docs/Extensibility_UG/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
+/docs/openvino-workflow/model-preparation/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ovc-maintainers
+/docs/openvino-workflow/running-inference/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
+/docs/openvino-extensibility/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
 /docs/snippets/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
-/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino_contrib-arm_plugin-maintainers
-/docs/OV_Runtime_UG/supported_plugins/CPU.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-cpu-maintainers
-/docs/OV_Runtime_UG/supported_plugins/GPU*.md @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gpu-maintainers
+/docs/articles_en/assets/snippets/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-maintainers
+/docs/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device.rst @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-cpu-maintainers
+/docs/openvino-workflow/running-inference/inference-devices-and-modes/cpu-device/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-cpu-maintainers
+/docs/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.rst @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gpu-maintainers
+/docs/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device/ @openvinotoolkit/openvino-docs-maintainers @openvinotoolkit/openvino-ie-gpu-maintainers
 
 # Configuration management
 /**/setup.py  @openvinotoolkit/openvino-configuration-mgmt

diff --git a/docs/articles_en/about-openvino/compatibility-and-support/supported-models.rst b/docs/articles_en/about-openvino/compatibility-and-support/supported-models.rst
@@ -18,14 +18,12 @@ models from OpenVINO-supported frameworks may also work properly but have not be
    :file:  ../../_static/download/supported_models.csv
 
 
-Check marks indicate models that passed inference with no errors. Empty cells indicate models
-that were not tested. No failing runs producing an error have been recorded.
-
-In the precision column, the "optimum-intel default" label corresponds to FP32 for small models
-and INT8 for models greater than 1B parameters.
-
-
-| Note:
+| Marked cells indicate models that passed inference with no errors. Empty cells indicate
+  models that were not tested. No failing runs producing an error have been recorded.
+|
+| In the precision column, the "optimum-intel default" label corresponds to FP32 for small models
+  and INT8 for models greater than 1B parameters.
+|
 | The results as of June 17 2024, for OpenVINO version 2024.2.
 | The models come from different public model repositories, such as Pytorch Model Zoo and
   HuggingFace; they were executed on the designated hardware with OpenVINO either natively or

diff --git a/...articles_en/about-openvino/performance-benchmarks/generative-ai-performance.rst b/...articles_en/about-openvino/performance-benchmarks/generative-ai-performance.rst
@@ -4,7 +4,7 @@ Most Efficient Large Language Models for AI PC
 This page is regularly updated to help you identify the best-performing LLMs on the
 Intel® Core™ Ultra processor family and AI PCs.
 
-The table below lists the key performance indicators for a selection of Large Language Models,
+The tables below list the key performance indicators for a selection of Large Language Models,
 running on an Intel® Core™ Ultra 7-165H based system, on built-in GPUs.
 
 
@@ -14,11 +14,47 @@ running on an Intel® Core™ Ultra 7-165H based system, on built-in GPUs.
    <label><link rel="stylesheet" type="text/css" href="../../_static/css/openVinoDataTables.css"></label>
 
 
-.. csv-table::
-   :class: modeldata stripe
-   :name: supportedModelsTable
-   :header-rows: 1
-   :file:  ../../_static/download/llm_models.csv
 
+.. tab-set::
 
-For complete information on the system config, see: `Hardware Platforms [PDF] <https://docs.openvino.ai/2024/_static/benchmarks_files/OV-2024.3-platform_list.pdf>`__
+   .. tab-item:: OpenVINO
+
+      .. csv-table::
+         :class: modeldata stripe
+         :name: supportedModelsTable
+         :header-rows: 1
+         :file:  ../../_static/download/llm_models.csv
+
+   .. tab-item:: OpenVINO Model Server
+
+      .. csv-table::
+         :class: modeldata stripe
+         :name: supportedModelsTable
+         :header-rows: 1
+         :file:  ../../_static/download/llm_models_ovms.csv
+
+
+
+For complete information on the system config, see:
+`Hardware Platforms [PDF] <https://docs.openvino.ai/2024/_static/benchmarks_files/OV-2024.3-platform_list.pdf>`__
+
+To view the data in an editable form, you can download the .csv files here:
+
+.. grid:: 1 1 2 2
+   :gutter: 4
+
+   .. grid-item::
+
+      .. button-link:: ../../_static/download/llm_models.csv
+         :color: primary
+         :outline:
+         :expand:
+
+         :material-regular:`download;1.5em` Click for OpenVINO LLM results [CSV]
+
+      .. button-link:: ../../_static/download/llm_models_ovms.csv
+         :color: primary
+         :outline:
+         :expand:
+
+         :material-regular:`download;1.5em` Click for OpenVINO Model Server results [CSV]
diff --git a/docs/articles_en/about-openvino/release-notes-openvino.rst b/docs/articles_en/about-openvino/release-notes-openvino.rst
@@ -116,12 +116,11 @@ NPU Device Plugin
 OpenVINO Python API
 -----------------------------
 
-* visit_attributes is now available in custom operation implemented in Python, so you may pass
-  a dictionary of attributes, i.e. {"name1": value1, "name2": value2...}, instead of multiple
-  on_attribute methods (as in C++).
-* ReadValue or NodeFactory can now be used to benefit different use cases, for reduced code
-  complexity.
-* Kwargs overloading is now supported.
+* visit_attributes is now available for custom operation implemented in Python, enabling
+  serialization of operation attributes.
+* Python API is now extended with new methods for Model class, e.g. Model.get_sink_index, new
+  overloads for Model.get_result_index.
+
 
 OpenVINO Node.js API
 -----------------------------

diff --git a/docs/sphinx_setup/_static/download/llm_models_ovms.csv b/docs/sphinx_setup/_static/download/llm_models_ovms.csv
@@ -0,0 +1,37 @@
+Product,Model,Framework,Precision,Node,Request Rate,Throughput [tok/s],TPOT Mean Latency
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8380,inf,270.55,839.37
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8380,2.0,269.6,847.81
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8380,1.0,268.92,840.1
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8380,0.2,92.75,75.75
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8480+,inf,702.42,307.82
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8480+,2.0,680.45,302.09
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8480+,1.0,442.69,169.24
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8480+,0.2,92.63,63.23
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8580,inf,701.91,305.9
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8580,2.0,684.4,299.41
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8580,1.0,442.46,170.65
+ovms,meta-llama/Llama-2-7b-chat-hf,PT,INT8-CW,Xeon Platinum 8580,0.2,92.89,54.69
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8380,inf,290.39,793.52
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8380,2.0,284.14,815.09
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8380,1.0,290.67,783.47
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8380,0.2,87.18,74.96
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8480+,inf,873.93,245.31
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8480+,2.0,774.3,233.49
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8480+,1.0,427.37,114.16
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8480+,0.2,88.9,60.04
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8580,inf,839.74,253.74
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8580,2.0,771.17,232.08
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8580,1.0,427.85,128.33
+ovms,mistralai/Mistral-7B-v0.1,PT,INT8-CW,Xeon Platinum 8580,0.2,88.92,56.33
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8380,inf,275.71,810.89
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8380,2.0,278.07,809.3
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8380,1.0,272.54,811.37
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8380,0.2,79.24,73.06
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8480+,inf,799.46,265.5
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8480+,2.0,707.46,234.44
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8480+,1.0,380.61,104.71
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8480+,0.2,78.3,60.37
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8580,inf,843.51,252.12
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8580,2.0,698.38,247.77
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8580,1.0,376.36,139.62
+ovms,meta-llama/Meta-Llama-3-8B-Instruct,PT,INT8-CW,Xeon Platinum 8580,0.2,78.61,54.12