Add PyTorch Frontend (openvinotoolkit#15069)

* WIP * update input validation * upsample_nearest2d and upsample_bilinear2d support * support leaky_relu add test for inplace relu * update tests, add handler for ListConstruct * Do not create extra outputs in main body * add positive case with non-default value * update testing * update test, handle non constant size and scale * remove ie_device * add aten::group_norm support * refactoring * Enable aten::reshape_as operator and add layer test * more tests * Fix typo in test * Resolve conflicts * fix code style * expand init version * expand_as and tests * add transposed convolutions support * add tests * initial support pad * add circular * update for differenced in rang * cleanup * refactor * more tests * apply review comments * Add split+listunpack transformation * Add split+getitem transformation * Add test cases * fix typo * Minor fixes * Apply suggestions from code review Co-authored-by: Maxim Vafin <[email protected]> * Apply suggestions from code review * Small fix * Support converting models without freezing * support BoolTensor and masked_fill * add support aten::rsqrt and test for sqrt * add cumsum and type_as * support clamp * support more matrix operations * add tests * Add aten::adaptive_avg_pool3d and layer test * Change to rank * fix code style in utils.hpp * Update src/frontends/pytorch/src/op_table.cpp Co-authored-by: Sergey Lyalin <[email protected]> * fix code style * add tests * add xfail * remove unnecessary broadcast * Changes required by style formater * aten::_convolution_mode * Changes requested by a reviewer * remove code duplication * add aten::unbind transformation * full, zeros and ones * Support getattr list and unrolling nested ifs * Remove line change * Enable back freezing in layer tests * Add aten::norm operator and layer test * Small fix in layer test * add aten::roll * add empty line * Typo fix * fix style * fix style v2 * add pytorch frontend to wheel * Support all types of numeric norms * add check for dynamic shapes * remove random change * merge statements * add min and max ops support * aten::max and aten::min * move axes range creation to utils * add transformation for tuple results, update tests * fix copyright * aten::var * add test and translation for numel * ignore aten::clone * Add layer test for aten::add operator * Fix typo * Remove redundant import * Add parameter name in forward method * fix code style * apply review comments * Add size+slice+listunpack transform * Add append listunpack transformation * Register transformation * aten::where * update realization * Fix issue with getitem * Fix getitem * Add layer test for aten::view operator * Add tests for listunpack * add test for aten::div * fix style * update aten::adaptive_max_pool2d * fix style * add aten::floor_divide * aten::addmm support alpha and beta with different dtype * nonzero * Change test name * update test cases to include other dtypes * aten::arange * prim::max transformation for ListConstruct * rename op * generalize conv2d implementation for conv1d and conv3d * aten::unsqueeze_ and tests for aten::unsqueeze (#70) * add aten::le, aten::ge and tests for other tensor comparision ops (#74) * add support trigonometry ops (#73) * support aten::upsample_bicubic2d, aten::ceil, aten::floor (#72) Co-authored-by: Maxim Vafin <[email protected]> * extend and add tests for avg_pool and max_pool * extend tests and constant filling ops * fix as_tensor and full ops * aten::repeat * fix code style * aten::im2col (#61) * aten::im2col * remove debug prints, add number of elements check * fix failed tests * move helper function * use split * Update src/frontends/pytorch/src/op/im2col.cpp Co-authored-by: Maxim Vafin <[email protected]> * fix code style Co-authored-by: Maxim Vafin <[email protected]> * Update src/frontends/pytorch/src/utils.cpp Co-authored-by: Maxim Vafin <[email protected]> * fix code style * revert removeinf floordiv, add floor_divide file * Fix merge issue * reduce code duplication * refactor * Add len operator with layer test * update clamp to support mixed precision and add support torch.long for constants * aten::selu * add trunc mode to div * add else statement * Add test case to layer test * Fix submodules (#88) * update test file * fix namings * execute in fp64 and convert back to initial precision * Revert set_output_size to master. Small fix in If validate * Fix build and code style * fix failed tests * Add torchvision::nms operator and layer test * Change requested by a reviewer * Remove div test * convert constants to input type * Mark some cases in div tests as xfail (#93) * Small refactoring (#94) * Small refactoring * Fix type * Fix python codestyle * Incremental fix code style (#95) * Fix style (#96) * Fix copyright * Fix code style * Branch clean up (#97) * Optimize includes and force opset10 (#98) * Optimize includes * Force opset10 in pt fe * Fix codestyle (#99) * Fix style * Fix clang codestyle * Fix cerr with debug log * Update src/bindings/python/src/pyopenvino/frontend/pytorch/decoder.cpp * Add pytorch dependency only if pytorch frontend is enabled * Update src/bindings/python/src/pyopenvino/CMakeLists.txt * Add layer tests to precommit (#100) * Add layer tests to precommit * Remove accidentally added files * Apply code style on layer tests * batch norm tests and fixes * move default weight and bias to else block * reduce code duplication * Changes requested by a reviewer * Changes requested by a reviewer * Remove dependency from pytorch in pyopenvino (#102) * Remove dependency from pytorch when fe is disabled * Change docstring * Remove pytorch FE dependency from pyopenvino * Apply codestyle (#107) * Apply codestyle * Remove commented line * Apply suggestions from code review Co-authored-by: Roman Kazantsev <[email protected]> * Fix mock FE test (#108) * Fix mock PE test (#111) * Revert changes in StridedSlice (#114) * Small refactoring (#116) * Small refactoring * Fix codestyle * Apply suggestions from code review Co-authored-by: Roman Kazantsev <[email protected]> * Apply suggestions from code review * Update src/frontends/pytorch/src/op/group_norm.cpp * Fix cmake copyright define (#117) * Update src/frontends/pytorch/src/op/arange.cpp * Apply suggestions from code review * Update build configs (#120) * FIx build configs * Update type cast in full.cpp * Apply review feedback (#121) * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Roman Kazantsev <[email protected]> * Fix issue after master merge (#122) * Fix issue after master merge * Fix build Co-authored-by: eaidova <[email protected]> Co-authored-by: bszmelcz <[email protected]> Co-authored-by: Sergey Lyalin <[email protected]> Co-authored-by: sikorsl1 <[email protected]> Co-authored-by: Leonard Sikorski <[email protected]> Co-authored-by: Mateusz <[email protected]> Co-authored-by: Roman Kazantsev <[email protected]>
a-sidorova · Jan 18, 2023 · 53e699e · 53e699e
1 parent 1794fb4
commit 53e699e
Show file tree

Hide file tree

Showing 203 changed files with 10,707 additions and 23 deletions.
diff --git a/.ci/azure/linux.yml b/.ci/azure/linux.yml
@@ -462,6 +462,13 @@ jobs:
       WORKSPACE: $(INSTALL_DIR)
     displayName: 'Samples Smoke Tests'
 
+  - script: |
+      python3 -m pip install -r $(LAYER_TESTS_DIR)/requirements.txt
+      export PYTHONPATH=$(REPO_DIR)/tools/mo/:$(LAYER_TESTS_DIR):$PYTHONPATH
+      export TEST_DEVICE=CPU
+      $(RUN_PREFIX) python3 -m pytest $(LAYER_TESTS_DIR)/pytorch_tests/ -m precommit --junitxml=$(INSTALL_TEST_DIR)/TEST-pytorch.xmlTEST
+    displayName: 'PyTorch Layer Tests'
+
   - script: |
       python3 -m pip install -r $(LAYER_TESTS_DIR)/requirements.txt
       export PYTHONPATH=$(REPO_DIR)/tools/mo/:$(LAYER_TESTS_DIR):$PYTHONPATH

diff --git a/.ci/azure/linux_conditional_compilation.yml b/.ci/azure/linux_conditional_compilation.yml
@@ -124,6 +124,7 @@ jobs:
         -DENABLE_TEMPLATE=OFF
         -DENABLE_OV_ONNX_FRONTEND=OFF
         -DENABLE_OV_PADDLE_FRONTEND=OFF
+        -DENABLE_OV_PYTORCH_FRONTEND=OFF
         -DENABLE_OV_TF_FRONTEND=OFF
         -S $(REPO_DIR)
         -B $(BUILD_DIR)

diff --git a/.ci/azure/linux_cuda.yml b/.ci/azure/linux_cuda.yml
@@ -135,6 +135,7 @@ jobs:
             -DENABLE_INTEL_GNA=OFF \
             -DENABLE_OV_TF_FRONTEND=OFF \
             -DENABLE_OV_PADDLE_FRONTEND=OFF \
+            -DENABLE_OV_PYTORCH_FRONTEND=OFF \
             -DENABLE_OV_ONNX_FRONTEND=OFF \
             -DENABLE_PYTHON=OFF \
             -DENABLE_TESTS=ON \

diff --git a/.ci/azure/linux_onnxruntime.yml b/.ci/azure/linux_onnxruntime.yml
@@ -115,6 +115,7 @@ jobs:
         -DENABLE_COMPILE_TOOL=OFF
         -DENABLE_OV_TF_FRONTEND=OFF
         -DENABLE_OV_PADDLE_FRONTEND=OFF
+        -DENABLE_OV_PYTORCH_FRONTEND=OFF
         -DENABLE_OPENVINO_DEBUG=OFF
         -S $(REPO_DIR)
         -B $(BUILD_DIR)

diff --git a/.ci/azure/windows_conditional_compilation.yml b/.ci/azure/windows_conditional_compilation.yml
@@ -130,6 +130,7 @@ jobs:
         -DENABLE_TESTS=OFF ^
         -DENABLE_OV_ONNX_FRONTEND=OFF ^
         -DENABLE_OV_PADDLE_FRONTEND=OFF ^
+        -DENABLE_OV_PYTORCH_FRONTEND=OFF ^
         -DENABLE_OV_TF_FRONTEND=OFF ^
         $(REPO_DIR)
     workingDirectory: $(BUILD_DIR)
@@ -175,6 +176,7 @@ jobs:
         -DENABLE_TESTS=OFF ^
         -DENABLE_OV_ONNX_FRONTEND=OFF ^
         -DENABLE_OV_PADDLE_FRONTEND=OFF ^
+        -DENABLE_OV_PYTORCH_FRONTEND=OFF ^
         -DENABLE_OV_TF_FRONTEND=OFF ^
         $(REPO_DIR)
     workingDirectory: $(BUILD_DIR_2)

diff --git a/.ci/openvino-onnx/Dockerfile b/.ci/openvino-onnx/Dockerfile
@@ -61,6 +61,7 @@ RUN cmake .. \
     -DENABLE_PROFILING_ITT=OFF \
     -DENABLE_SAMPLES=OFF \
     -DENABLE_OV_PADDLE_FRONTEND=OFF \
+    -DENABLE_OV_PYTORCH_FRONTEND=OFF \
     -DENABLE_OV_TF_FRONTEND=OFF \
     -DENABLE_OPENVINO_DEBUG=OFF \
     -DCMAKE_INSTALL_PREFIX=/openvino/dist

diff --git a/cmake/coverage.cmake b/cmake/coverage.cmake
@@ -136,6 +136,13 @@ if(ENABLE_OV_PADDLE_FRONTEND)
         PREFIX "${OV_COVERAGE_BASE_DIRECTORY}")
 endif()
 
+if(ENABLE_OV_PYTORCH_FRONTEND)
+    ov_coverage_extract(INPUT "openvino" OUTPUT "pytorch_frontend"
+        PATTERNS "${OV_COVERAGE_BASE_DIRECTORY}/src/frontends/pytorch/*")
+    ov_coverage_genhtml(INFO_FILE "pytorch_frontend"
+        PREFIX "${OV_COVERAGE_BASE_DIRECTORY}")
+endif()
+
 if(ENABLE_OV_TF_FRONTEND)
     ov_coverage_extract(INPUT "openvino" OUTPUT "tf_frontend"
         PATTERNS "${OV_COVERAGE_BASE_DIRECTORY}/src/frontends/tensorflow/*")

diff --git a/cmake/features.cmake b/cmake/features.cmake
@@ -151,6 +151,7 @@ ie_dependent_option (ENABLE_CPU_DEBUG_CAPS "enable CPU debug capabilities at run
 find_host_package(PythonInterp 3 QUIET)
 ie_option(ENABLE_OV_ONNX_FRONTEND "Enable ONNX FrontEnd" ${PYTHONINTERP_FOUND})
 ie_option(ENABLE_OV_PADDLE_FRONTEND "Enable PaddlePaddle FrontEnd" ON)
+ie_option(ENABLE_OV_PYTORCH_FRONTEND "Enable PyTorch FrontEnd" ON)
 ie_option(ENABLE_OV_TF_FRONTEND "Enable TensorFlow FrontEnd" ON)
 ie_dependent_option(ENABLE_SYSTEM_PROTOBUF "Use system protobuf" OFF
     "ENABLE_OV_ONNX_FRONTEND OR ENABLE_OV_PADDLE_FRONTEND OR ENABLE_OV_TF_FRONTEND;BUILD_SHARED_LIBS" OFF)

diff --git a/cmake/packaging/debian.cmake b/cmake/packaging/debian.cmake
@@ -261,6 +261,19 @@ macro(ov_cpack_settings)
         set(paddle_copyright "generic")
     endif()
 
+    if(ENABLE_OV_PYTORCH_FRONTEND)
+        set(CPACK_COMPONENT_PYTORCH_DESCRIPTION "OpenVINO PyTorch Frontend")
+        set(CPACK_COMPONENT_PYTORCH_DEPENDS "${OV_CPACK_COMP_CORE}")
+        set(CPACK_DEBIAN_PYTORCH_PACKAGE_NAME "libopenvino-pytorch-frontend-${cpack_name_ver}")
+        # since we PYTORCH FE is linkable target, we need to call ldconfig (i.e. `def_triggers`)
+        set(CPACK_DEBIAN_PYTORCH_PACKAGE_CONTROL_EXTRA "${def_postinst};${def_postrm};${def_triggers}")
+        ov_debian_add_lintian_suppression(pytorch
+            # we have different package name strategy; it suggests libopenvino-pytorch-frontend202230
+            "package-name-doesnt-match-sonames")
+        list(APPEND frontends pytorch)
+        set(pytorch_copyright "generic")
+    endif()
+
     #
     # core_dev: depends on core and frontends (since frontends don't want to provide its own dev packages)
     #

diff --git a/cmake/packaging/rpm.cmake b/cmake/packaging/rpm.cmake
@@ -226,6 +226,15 @@ macro(ov_cpack_settings)
         set(paddle_copyright "generic")
     endif()
 
+    if(ENABLE_OV_PYTORCH_FRONTEND)
+        set(CPACK_COMPONENT_PYTORCH_DESCRIPTION "OpenVINO PyTorch Frontend")
+        set(CPACK_RPM_PYTORCH_PACKAGE_NAME "libopenvino-pytorch-frontend-${cpack_name_ver}")
+        set(CPACK_RPM_PYTORCH_POST_INSTALL_SCRIPT_FILE "${def_triggers}")
+        set(CPACK_RPM_PYTORCH_POST_UNINSTALL_SCRIPT_FILE "${def_triggers}")
+        _ov_add_package(frontend_packages pytorch)
+        set(pytorch_copyright "generic")
+    endif()
+
     #
     # core_dev: depends on core and frontends (since frontends don't want to provide its own dev packages)
     #

diff --git a/cmake/templates/OpenVINOConfig.cmake.in b/cmake/templates/OpenVINOConfig.cmake.in
@@ -12,6 +12,7 @@
 #  * `Runtime`: OpenVINO C++ and C Core & Inference Runtime, frontend common
 #  * `ONNX`: OpenVINO ONNX frontend
 #  * `Paddle`: OpenVINO Paddle frontend
+#  * `PyTorch`: OpenVINO PyTorch frontend
 #  * `TensorFlow`: OpenVINO TensorFlow frontend
 #
 # If no components are specified, `Runtime` component is provided:
@@ -41,6 +42,9 @@
 #   `openvino::frontend::paddle`
 #   Paddle FrontEnd target (optional)
 #
+#   `openvino::frontend::pytorch`
+#   PyTorch FrontEnd target (optional)
+#
 #   `openvino::frontend::tensorflow`
 #   TensorFlow FrontEnd target (optional)
 #
@@ -61,6 +65,9 @@
 #   `OpenVINO_Frontend_Paddle_FOUND`
 #   OpenVINO Paddle frontend is available
 #
+#   `OpenVINO_Frontend_PyTorch_FOUND`
+#   OpenVINO PyTorch frontend is available
+#
 #   `OpenVINO_Frontend_TensorFlow_FOUND`
 #   OpenVINO TensorFlow frontend is available
 #
@@ -293,11 +300,13 @@ set(${CMAKE_FIND_PACKAGE_NAME}_ONNX_FOUND @ENABLE_OV_ONNX_FRONTEND@)
 set(${CMAKE_FIND_PACKAGE_NAME}_Paddle_FOUND @ENABLE_OV_PADDLE_FRONTEND@)
 set(${CMAKE_FIND_PACKAGE_NAME}_TensorFlow_FOUND @ENABLE_OV_TF_FRONTEND@)
 set(${CMAKE_FIND_PACKAGE_NAME}_IR_FOUND @ENABLE_OV_IR_FRONTEND@)
+set(${CMAKE_FIND_PACKAGE_NAME}_PyTorch_FOUND @ENABLE_OV_PYTORCH_FRONTEND@)
 
 set(${CMAKE_FIND_PACKAGE_NAME}_Frontend_ONNX_FOUND ${${CMAKE_FIND_PACKAGE_NAME}_ONNX_FOUND})
 set(${CMAKE_FIND_PACKAGE_NAME}_Frontend_Paddle_FOUND ${${CMAKE_FIND_PACKAGE_NAME}_Paddle_FOUND})
 set(${CMAKE_FIND_PACKAGE_NAME}_Frontend_TensorFlow_FOUND ${${CMAKE_FIND_PACKAGE_NAME}_TensorFlow_FOUND})
 set(${CMAKE_FIND_PACKAGE_NAME}_Frontend_IR_FOUND ${${CMAKE_FIND_PACKAGE_NAME}_IR_FOUND})
+set(${CMAKE_FIND_PACKAGE_NAME}_Frontend_PyTorch_FOUND ${${CMAKE_FIND_PACKAGE_NAME}_PyTorch_FOUND})
 
 # if no components specified, only Runtime is provided
 if(NOT ${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS)

diff --git a/src/bindings/python/src/openvino/frontend/pytorch/__init__.py b/src/bindings/python/src/openvino/frontend/pytorch/__init__.py
@@ -0,0 +1,21 @@
+# Copyright (C) 2018-2023 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+Package: openvino
+Low level wrappers for the FrontEnd C++ API.
+"""
+
+# flake8: noqa
+
+from openvino.utils import add_openvino_libs_to_path
+
+add_openvino_libs_to_path()
+
+
+try:
+    from openvino.frontend.pytorch.py_pytorch_frontend import _FrontEndPytorchDecoder as Decoder
+    from openvino.frontend.pytorch.py_pytorch_frontend import _Type as DecoderType
+except ImportError as err:
+    raise ImportError("OpenVINO PyTorch frontend is not available, please make sure the frontend is built."
+                      "{}".format(err))