Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Torch compile/new op support - Test PR #34

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
381b7ee
[DOCS) install guide tweaks (#22976)
kblaszczak-intel Feb 23, 2024
632174b
DOCS Udpate banchmark files (#23029)
akopytko Feb 23, 2024
1e5d0e5
[op conformance] Fix threshold bugs in ov::test::utils::compare (#22972)
sbalandi Feb 23, 2024
c768c0e
Updated Gen.AI and NNCF documentation (#22793)
AlexKoff88 Feb 23, 2024
ac4efa5
fix #18388 by recreate `ov::Model` using the main runtime and assign …
FredBill1 Feb 23, 2024
8440051
Fix register conflict in jit_avx512_dw_f32 primitive (#22693)
mangguo321 Feb 23, 2024
05b0fca
[PyOV] Add doc strings to Python Remote API functions (#23030)
Feb 23, 2024
5f64019
Support OpExtension in frontends (#22961)
mvafin Feb 23, 2024
ac04160
[CPU] Fixed BF16 Matmul inference precision (#22994)
dmitry-gorokhov Feb 23, 2024
1686e03
refactoring get_cpu_pinning() (#22489)
wangleis Feb 23, 2024
bdf6fce
Replace `func_test_utils` with `common` in `ov_proxy_plugin_tests` (#…
vurusovs Feb 23, 2024
d58efea
Remove `ngraph::helper` (#22704)
vurusovs Feb 23, 2024
09956e1
[OVC] Removed paths checks from OVC workflow. (#23056)
popovaan Feb 23, 2024
f9605cd
[TF FE] Add layer test for tf.raw_ops.RaggedTensorToTensor (#23041)
rkazants Feb 23, 2024
c40b675
[GPU] Fix FC shape agnostic kernel build failure issue. (#22868)
hyunback Feb 26, 2024
69736c6
[DOCS] Update model table for master (#23080)
akopytko Feb 26, 2024
fd451f5
[Layer Test] Remove legacy code from layer tests (#23067)
rkazants Feb 26, 2024
a68c820
[DOCS] Add Node.js API to docs (#23055)
sgolebiewski-intel Feb 26, 2024
a819605
[TF FE] Test TF Hugging Face models (#22970)
popovaan Feb 26, 2024
a5f6308
[DOCS] Update Interactive Tutorials with the selector tool (#23088)
sgolebiewski-intel Feb 26, 2024
089fc0d
[TF FE] MatrixBandPart operation for TensorFlow Hub models (#23082)
rkazants Feb 26, 2024
cd47ad5
Enable some fixed tests (#23057)
olpipi Feb 26, 2024
af5557f
revert PR 22489 (#23078)
wangleis Feb 27, 2024
ddf7c44
[GPU] Add a new option for DumpProfilingData (#23006)
andrew-k-park Feb 27, 2024
0ef2fab
[Snippets][Doc] Updated the design guide (#22258)
a-sidorova Feb 27, 2024
7ab6ec5
[AUTO BATCH] Clean legacy name (#22909)
zhaixuejun1993 Feb 27, 2024
1f88f2e
[Snippets] Specific loop iterations handler (#21303)
v-Golubev Feb 27, 2024
b4d26b6
[CPU] Moved Ngram fusion with SymbolicOpts before Snippets pipeline (…
a-sidorova Feb 27, 2024
b600e97
[CPU] Updated aux GPR count for Load/Store emitters (#23046)
a-sidorova Feb 27, 2024
0474cf7
[CPU] Introduce gemm conv post op checks (#23059)
maxnick Feb 27, 2024
b711ed0
[CONFORMANCE] Fix core_threading test on Template + reusing core in a…
iefode Feb 27, 2024
c05981d
[PT FE] Support multiple FX operations
mvafin Feb 27, 2024
b3a3f14
Apply codestyle
mvafin Feb 27, 2024
45f4a3b
Apply suggestions from code review
mvafin Feb 27, 2024
10bb014
Fix tests
mvafin Feb 27, 2024
7aa6cd6
New ops added to torch.compile support list
cavusmustafa Feb 28, 2024
9f4bdae
Additional ops for NetVLad and ALIKE
cavusmustafa Feb 28, 2024
272e0f2
Additional op support for ChatGLM2
cavusmustafa Feb 28, 2024
620bec2
PTFE input_model constructor output access fix
cavusmustafa Feb 28, 2024
957eb29
Removed bitwise_not fx version
cavusmustafa Feb 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
/tests/layer_tests/tensorflow_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/layer_tests/jax_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests @openvinotoolkit/openvino-tf-frontend-maintainers
/tests/model_hub_tests/torch_tests @openvinotoolkit/openvino-pytorch-frontend-maintainers
/tests/model_hub_tests/pytorch @openvinotoolkit/openvino-pytorch-frontend-maintainers

# Tools:
/tools/ @openvinotoolkit/openvino-tools-maintainers
Expand Down
4 changes: 2 additions & 2 deletions .github/labeler.yml
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@
- 'tests/layer_tests/tensorflow2_keras_tests/**/*'
- 'tests/layer_tests/jax_tests/**/*'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/torch_tests/**/*']
'!tests/model_hub_tests/pytorch/**/*']

'category: TFL FE':
- 'src/frontends/tensorflow_lite/**/*'
Expand All @@ -156,7 +156,7 @@
- 'tests/layer_tests/py_frontend_tests/test_torch_decoder.py'
- 'tests/layer_tests/py_frontend_tests/test_torch_frontend.py'
- any: ['tests/model_hub_tests/**',
'!tests/model_hub_tests/tf_hub_tests/**/*']
'!tests/model_hub_tests/tensorflow/**/*']

'category: tools':
- any: ['tools/**',
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/job_python_unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ jobs:
- name: PyTorch torch.export Layer Tests
if: ${{ fromJSON(inputs.affected-components).PyTorch_FE.test && runner.arch != 'ARM64' }} # Ticket: 126287
run: |
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -m precommit_torch_export --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
python3 -m pytest ${LAYER_TESTS_INSTALL_DIR}/pytorch_tests -n logical -m precommit_torch_export --junitxml=${INSTALL_TEST_DIR}/TEST-pytorch.xml
env:
TEST_DEVICE: CPU
TEST_PRECISION: FP32
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/job_pytorch_models_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ jobs:

- name: Install PyTorch tests requirements
run: |
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/requirements.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/requirements_secondary.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/requirements.txt
python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/requirements_secondary.txt
echo "Available storage:"
df -h
env:
Expand All @@ -121,7 +121,7 @@ jobs:
- name: PyTorch Models Tests
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_model_tests.html --self-contained-html -v
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-torch_model_tests.html --self-contained-html -v
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
Expand All @@ -131,7 +131,7 @@ jobs:
- name: Reformat unsupported ops file
if: '!cancelled()'
run: |
python3 ${MODEL_HUB_TESTS_INSTALL_DIR}/torch_tests/scripts/process_op_report.py ${INSTALL_TEST_DIR}/TEST-torch_unsupported_ops.log
python3 ${MODEL_HUB_TESTS_INSTALL_DIR}/pytorch/scripts/process_op_report.py ${INSTALL_TEST_DIR}/TEST-torch_unsupported_ops.log

- name: Available storage after tests
run: |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: TensorFlow Hub Models tests
name: TensorFlow Models tests

on:
workflow_call:
Expand Down Expand Up @@ -109,12 +109,12 @@ jobs:
python3 -m pip install ${INSTALL_DIR}/openvino_tokenizers-*

- name: Install TF Hub tests requirements
run: python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/tf_hub_tests/requirements.txt
run: python3 -m pip install -r ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/requirements.txt

- name: TensorFlow Hub Tests - TF FE
- name: TensorFlow Models Tests - TF FE
run: |
export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tf_hub_tests/ -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-tf_hub_tf_fe.html --self-contained-html -v
python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/tensorflow/ -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-tf_hub_tf_fe.html --self-contained-html -v
env:
TYPE: ${{ inputs.event == 'schedule' && 'nightly' || 'precommit'}}
TEST_DEVICE: CPU
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -501,12 +501,12 @@ jobs:
# image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
# volumes:
# - /mount:/mount
TensorFlow_Hub_Models_Tests:
name: TensorFlow Hub Models tests
TensorFlow_Models_Tests:
name: TensorFlow Models tests
if: fromJSON(needs.smart_ci.outputs.affected_components).TF_FE.test ||
fromJSON(needs.smart_ci.outputs.affected_components).TFL_FE.test
needs: [ Build, Smart_CI, Openvino_tokenizers ]
uses: ./.github/workflows/job_tensorflow_hub_models_tests.yml
uses: ./.github/workflows/job_tensorflow_models_tests.yml
with:
runner: ${{ github.event_name == 'schedule' && 'ubuntu-20.04-16-cores' || 'ubuntu-20.04-8-cores' }}
event: ${{ github.event_name }}
Expand Down Expand Up @@ -755,7 +755,7 @@ jobs:
Overall_Status:
name: ci/gha_overall_status
needs: [Smart_CI, Build, Debian_Packages, Samples, Conformance, ONNX_Runtime, CXX_Unit_Tests, Python_Unit_Tests,
CPU_Functional_Tests, TensorFlow_Hub_Models_Tests, PyTorch_Models_Tests, NVIDIA_Plugin, Openvino_tokenizers]
CPU_Functional_Tests, TensorFlow_Models_Tests, PyTorch_Models_Tests, NVIDIA_Plugin, Openvino_tokenizers]
if: ${{ always() }}
runs-on: ubuntu-latest
steps:
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/linux_arm64.yml
Original file line number Diff line number Diff line change
Expand Up @@ -361,13 +361,13 @@ jobs:
runner: 'aks-linux-16-cores-arm'
image: 'openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04'

TensorFlow_Hub_Models_Tests:
name: TensorFlow Hub Models tests
TensorFlow_Models_Tests:
name: TensorFlow Models tests
if: ${{ 'false' }} # TODO: Enable once the dependencies are ready for arm (no tensorflow-text available for arm from PyPI)
# if: fromJSON(needs.smart_ci.outputs.affected_components).TF_FE.test ||
# fromJSON(needs.smart_ci.outputs.affected_components).TFL_FE.test
needs: [ Build, Smart_CI ]
uses: ./.github/workflows/job_tensorflow_hub_models_tests.yml
uses: ./.github/workflows/job_tensorflow_models_tests.yml
with:
runner: 'aks-linux-16-cores-arm'
container: '{"image": "openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04"}'
Expand All @@ -387,7 +387,7 @@ jobs:
Overall_Status:
name: ci/gha_overall_status_linux_arm64
needs: [Smart_CI, Build, Debian_Packages, Samples, ONNX_Runtime, CXX_Unit_Tests, Python_Unit_Tests, CPU_Functional_Tests,
TensorFlow_Hub_Models_Tests, PyTorch_Models_Tests]
TensorFlow_Models_Tests, PyTorch_Models_Tests]
if: ${{ always() }}
runs-on: ubuntu-latest
steps:
Expand Down
6 changes: 0 additions & 6 deletions .github/workflows/ovc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,10 @@ name: OVC
on:
merge_group:
push:
paths:
- 'tools/ovc/**'
- '.github/workflows/ovc.yml'
branches:
- 'master'
- 'releases/**'
pull_request:
paths:
- 'tools/ovc/**'
- '.github/workflows/ovc.yml'

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand Down
4 changes: 2 additions & 2 deletions docs/articles_en/about-openvino/performance-benchmarks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -192,9 +192,9 @@ or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devc
Disclaimers
####################################

* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2023.3, as of January 16, 2024.
* Intel® Distribution of OpenVINO™ toolkit performance results are based on release 2023.3, as of February 13, 2024.

* OpenVINO Model Server performance results are based on release 2023.0, as of June 06, 2023.
* OpenVINO Model Server performance results are based on release 2023.3, as of February 13, 2024.

The results may not reflect all publicly available updates. Intel technologies’ features and benefits depend on system configuration
and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ Model Accuracy



The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
The following two tables present the absolute accuracy drop calculated as the accuracy difference
between OV-accuracy and the original frame work accuracy for FP32, and the same for INT8, BF16 and
FP16 representations of a model on three platform architectures. Please also refer to notes below
Expand Down Expand Up @@ -106,19 +109,19 @@ the table for more information.
- lambada openai
- ppl
-
- 17.37 (ref. 17.48)
- 17.41 (ref. 17.48)
- 17.16 (ref. 17.48)
- 17.38
- 17.41
- 17.17
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
- 3.24 (ref. 3.26)
- 3.24 (ref. 3.26)
- 3.25 (ref. 3.26)
- 3.24
- 3.24
- 3.25
* - Stable-Diffusion-V2-1
- LIAON-5B
- ppl
- CLIP
-
-
-
Expand All @@ -127,9 +130,10 @@ the table for more information.
- proprietary Mistral.ai
- ppl
-
- 3.29 (ref. 3.19)
- 3.28 (ref. 3.19)
-
- 3.29
- 3.47
- 3.49

.. list-table:: Model Accuracy for BF16, FP32 and FP16 (FP16: Flex-170 only. BF16: Xeon(R) 8490H only)
:header-rows: 1

Expand Down Expand Up @@ -233,36 +237,35 @@ the table for more information.
- lambada openai
- ppl
-
- 17.48 (ref. 17.48)
- 17.56 (ref. 17.48)
- 17.48
- 17.56
-
- 17.49 (ref. 17.48)
- 17.49
* - Llama-2-7b-chat
- Wiki, StackExch, Crawl
- ppl
-
- 3.26 (ref. 3.26)
- 3.26 (ref. 3.26)
- 3.26
- 3.26
-
-
* - Stable-Diffusion-V2-1
- LIAON-5B
- ppl
-
- CLIP
-
-
-
-
- 22.48
* - Mistral-7b
- proprietary Mistral.ai
- ppl
-
- 3.18 (ref. 3.19)
- 3.18 (ref. 3.19)
- 3.19
- 3.18
-
-

Notes: For all accuracy metrics except perplexity a "-", (minus sign), indicates an accuracy drop.
For perplexity (ppl) the values do not indicate a deviation from a reference but are the actual measured
accuracy for the model.

Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class that works well if all the following conditions are satisfied:

.. note::

``OpExtension`` class is currently available for ONNX and TensorFlow frontends.
``OpExtension`` class is currently available for ONNX, TensorFlow and PyTorch frontends.
PaddlePaddle frontend has named inputs and outputs for operation (not indexed)
therefore OpExtension mapping is not applicable for this case.

Expand Down
1 change: 0 additions & 1 deletion docs/articles_en/get-started/install-openvino-overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ Install OpenVINO™ 2024.0
OpenVINO Runtime on Linux <openvino_docs_install_guides_installing_openvino_linux_header>
OpenVINO Runtime on Windows <openvino_docs_install_guides_installing_openvino_windows_header>
OpenVINO Runtime on macOS <openvino_docs_install_guides_installing_openvino_macos_header>
Create a Yocto Image <openvino_docs_install_guides_installing_openvino_yocto>


.. raw:: html
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ To delete the files:
Additional Resources
####################

* :ref:`Troubleshooting Guide for OpenVINO Installation & Configuration <troubleshooting guide for install>`
* :doc:`Troubleshooting Guide for OpenVINO Installation & Configuration <openvino_docs_install_guides_overview>`
* Converting models for use with OpenVINO™: :ref:`Model Optimizer User Guide <deep learning model optimizer>`
* Writing your own OpenVINO™ applications: :ref:`OpenVINO™ Runtime User Guide <deep learning openvino runtime>`
* Sample applications: :ref:`OpenVINO™ Toolkit Samples Overview <code samples>`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ Use either of the following methods to delete the files:
Additional Resources
####################

* :ref:`Troubleshooting Guide for OpenVINO Installation & Configuration <troubleshooting guide for install>`
* :doc:`Troubleshooting Guide for OpenVINO Installation & Configuration <openvino_docs_install_guides_overview>`
* Converting models for use with OpenVINO™: :ref:`Model Optimizer Developer Guide <deep learning model optimizer>`
* Writing your own OpenVINO™ applications: :ref:`OpenVINO™ Runtime User Guide <deep learning openvino runtime>`
* Sample applications: :ref:`OpenVINO™ Toolkit Samples Overview <code samples>`
Expand Down
Loading
Loading