diff --git a/ngraph/python/BUILDING.md b/ngraph/python/BUILDING.md index 98ad3be60f1e0d..5d4d9c72892b3c 100644 --- a/ngraph/python/BUILDING.md +++ b/ngraph/python/BUILDING.md @@ -1,45 +1,44 @@ -# Building the Python API for nGraph +# Building the nGraph Python* API -You can build the nGraph Python API from sources by following instructions in this document. A Python wheel is a -portable package which will allow you to install nGraph in your Python distribution, or dedicated virtual environment. +This document provides the instructions for building the nGraph Python API from source on Linux, macOS and Windows 10 platforms. -## Build nGraph Python Wheels on Linux or MacOS +For each platform, you can build and install the API as a part of OpenVINO™ Toolkit or as a Python wheel. +A Python wheel is a portable package that allows you to install nGraph in your Python distribution, or dedicated virtual environment. + +## Linux* and macOS* ### Prerequisites -In order to build the nGraph Python wheel, you will need to install a few packages. +To build the nGraph Python API, you need to install a few additional packages. -On Ubuntu 20.04 LTS you can use the following instructions to install the required packages, including Python and Cython. +On Ubuntu* 20.04 LTS you can use the following instructions to install the required packages, including Python and Cython. apt install git wget build-essential cmake apt install python3 python3-dev python3-pip python3-virtualenv python-is-python3 -You can see a full working example on an Ubuntu environment used in our continuous environment in this -[Dockerfile](https://github.com/openvinotoolkit/openvino/blob/master/.ci/openvino-onnx/Dockerfile). - -On MacOS you can use [Homebrew](https://brew.sh) to install required packages: +On macOS, you can use [Homebrew](https://brew.sh) to install required packages: brew install cmake brew install automake brew install libtool brew install python3 -Install Cython in the Python installation, or virtualenv which you are planning to use: +Install Cython in the Python installation, or virtualenv that you are planning to use: pip3 install cython -### Configure, build and install OpenVINO + ### Configure and Build as a part of OpenVINO™ Toolkit on Linux and macOS -The following section will illustrate how to download, build and install OpenVINO in a workspace directory specified -by the `${MY_OPENVINO_BASEDIR}` variable. Let's start by setting this variable to a directory of your choice: +The following section illustrates how to build and install OpenVINO™ in a workspace directory using CMake. +The workspace directory is specified by the `${OPENVINO_BASEDIR}` variable. Set this variable to a directory of your choice: - export MY_OPENVINO_BASEDIR=/path/to/my/workspace + export OPENVINO_BASEDIR=/path/to/my/workspace -Now we can clone OpenVINO, configure it using `cmake` and build using `make`. Please note that we're disabling -the building of a few modules by setting the `ENABLE_*` flag to `OFF`. In order to build the OpenVINO Python APIs +Now you can clone the OpenVINO™ repository, configure it using `cmake` and build using `make`. Please note that we're disabling +the building of a few modules by setting the `ENABLE_*` flag to `OFF`. In order to build the OpenVINO™ Python APIs set the mentioned flags to `ON`. Note the `CMAKE_INSTALL_PREFIX`, which defaults to `/usr/local/` if not set. - cd "${MY_OPENVINO_BASEDIR}" + cd "${OPENVINO_BASEDIR}" git clone --recursive https://github.com/openvinotoolkit/openvino.git mkdir openvino/build cd openvino/build @@ -51,64 +50,64 @@ set the mentioned flags to `ON`. Note the `CMAKE_INSTALL_PREFIX`, which defaults -DENABLE_PYTHON=ON \ -DNGRAPH_PYTHON_BUILD_ENABLE=ON \ -DNGRAPH_ONNX_IMPORT_ENABLE=ON \ - -DCMAKE_INSTALL_PREFIX="${MY_OPENVINO_BASEDIR}/openvino_dist" + -DCMAKE_INSTALL_PREFIX="${OPENVINO_BASEDIR}/openvino_dist" make -j 4 make install -If you would like to use a specific version of Python, or use a virtual environment you can set the `PYTHON_EXECUTABLE` -variable. Examples: +The Python module is installed in the `${OPENVINO_BASEDIR}/openvino_dist/python/python/` folder. +Set up the OpenVINO™ environment in order to add the module path to `PYTHONPATH`: + + source ${OPENVINO_BASEDIR}/openvino_dist/bin/setupvars.sh + +If you would like to use a specific version of Python, or use a virtual environment, you can set the `PYTHON_EXECUTABLE` +variable. For example: ``` -DPYTHON_EXECUTABLE=/path/to/venv/bin/python -DPYTHON_EXECUTABLE=$(which python3.8) -``` +``` -### Build nGraph Python wheel +### Build an nGraph Python Wheel on Linux and macOS -When OpenVINO is built and installed, we can build the Python wheel by issuing the following command: +You can build the Python wheel running the following command: - make python_wheel + cd "${OPENVINO_BASEDIR}/openvino/ngraph/python" + python3 setup.py bdist_wheel Once completed, the wheel package should be located under the following path: - $ ls "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python/dist/" + $ ls "${OPENVINO_BASEDIR}/openvino/ngraph/python/dist/" ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl You can now install the wheel in your Python environment: - cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python/dist/" + cd "${OPENVINO_BASEDIR}/openvino/ngraph/python/dist/" pip3 install ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl -#### What does `make python_wheel` do? - -The `python_wheel` target automates a few steps, required to build the wheel package. You can recreate the process -manually by issuing the following commands: - - cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python" - git clone --branch v2.5.0 https://github.com/pybind/pybind11.git - export NGRAPH_CPP_BUILD_PATH="${MY_OPENVINO_BASEDIR}/openvino_dist" - python3 setup.py bdist_wheel - - -## Build nGraph Python Wheels on Windows +## Windows* 10 ### Prerequisites -In order to build OpenVINO and the nGraph Python wheel on Windows, you will need to install Visual Studio and Python. +In order to build OpenVINO™ and the nGraph Python wheel on Windows, you need to install Microsoft Visual Studio* and Python. + +Once Python is installed, you also need to install Cython using `pip install cython`. -Once Python is installed, you will also need to install Cython using `pip install cython`. +### Configure and Build as a Part of OpenVINO™ Toolkit on Windows -### Configure, build and install OpenVINO +The following section illustrates how to build and install OpenVINO™ in a workspace directory using CMake. +The workspace directory is specified by the `OPENVINO_BASEDIR` variable. Set this variable to a directory of your choice: + + set OPENVINO_BASEDIR=/path/to/my/workspace -Configure the build with a `cmake` invocation similar to the following. Note that you'll need to set the `-G` and -`-DCMAKE_CXX_COMPILER` to match the version and location of your Visual Studio installation. +Configure the build with a `cmake` invocation similar to the following. Note that need to set `-G` and +`-DCMAKE_CXX_COMPILER` to match the version and location of your Microsoft Visual Studio installation. ``` cmake .. ^ -G"Visual Studio 16 2019" ^ -DCMAKE_BUILD_TYPE=Release ^ - -DCMAKE_INSTALL_PREFIX="C:\temporary_install_dir" ^ + -DCMAKE_INSTALL_PREFIX="%OPENVINO_BASEDIR%/openvino_dist" ^ -DENABLE_CLDNN=OFF ^ -DENABLE_OPENCV=OFF ^ -DENABLE_VPU=OFF ^ @@ -120,14 +119,9 @@ cmake .. ^ ``` There are a couple of things to notice here. One is that the full path to the x64 version of -MSVC compiler has to be specified. This is because DNNL requires a 64-bit version and cmake may +MSVC compiler has to be specified. This is because DNNL requires a 64-bit version and `cmake` may fail to detect it correctly. -The other equally important thing to note is that the temporary directory where the build is to be installed can be specified. -If the installation directory is not specified, the default location is `C:\Program Files\OpenVINO`. -This examples uses `C:\temporary_install_dir` however, a subdirectory of `openvino\build` works as well. -The final Python wheel will contain the contents of this temporary directory so it's important to set it. - If you want to specify an exact Python version, use the following options: ``` -DPYTHON_EXECUTABLE="C:\Program Files\Python37\python.exe" ^ @@ -135,26 +129,32 @@ If you want to specify an exact Python version, use the following options: -DPYTHON_INCLUDE_DIR="C:\Program Files\Python37\include" ^ ``` -In order to build and install OpenVINO, build the `install` target: +In order to build and install OpenVINO™, build the `install` target: cmake --build . --target install --config Release -j 8 -In this step OpenVINO will be built and installed to the directory specified above. You can +In this step, OpenVINO™ is built and installed to the directory specified above. You can adjust the number of threads used in the building process to your machine's capabilities. +Set up the OpenVINO™ environment in order to add a module path to `PYTHONPATH`: + + %OPENVINO_BASEDIR%\openvino_dist\bin\setupvars.bat + +### Build an nGraph Python Wheel on Windows + Build the Python wheel package: - cmake --build . --target python_wheel --config Release -j 8 + cd "%OPENVINO_BASEDIR%/openvino/ngraph/python" + python setup.py bdist_wheel -The final wheel should be located in `ngraph\python\dist` directory. +The final wheel should be located in the `ngraph\python\dist` directory. dir openvino\ngraph\python\dist\ 10/09/2020 04:06 PM 4,010,943 ngraph_core-0.0.0-cp38-cp38-win_amd64.whl +## Run Tests -## Run tests - -### Using a virtualenv (optional) +### Use a virtualenv (Optional) You may wish to use a virutualenv for your installation. @@ -162,25 +162,20 @@ You may wish to use a virutualenv for your installation. $ source venv/bin/activate (venv) $ -### Install the nGraph wheel and other requirements +### Install the nGraph Wheel and Other Requirements - (venv) $ cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python" + (venv) $ cd "${OPENVINO_BASEDIR}/openvino/ngraph/python" (venv) $ pip3 install -r requirements.txt (venv) $ pip3 install -r requirements_test.txt (venv) $ pip3 install dist/ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl -### Run tests +### Run Tests You should now be able to run tests. -You may need to run the `setupvars` script from OpenVINO to set paths to OpenVINO components. - - source ${MY_OPENVINO_BASEDIR}/openvino/scripts/setupvars/setupvars.sh - -The minimum requirement is to set the `PYTHONPATH` to include the Inference Engine Python API: +You may need to run the `setupvars` script from the OpenVINO™ Toolkit to set paths to OpenVINO™ components. - export PYTHONPATH="${MY_OPENVINO_BASEDIR}/openvino/bin/intel64/Release/lib/python_api/python3.8/":${PYTHONPATH} - pytest tests/ + source ${OPENVINO_BASEDIR}/openvino_dist/bin/setupvars.sh Now you can run tests using `pytest`: diff --git a/ngraph/python/CMakeLists.txt b/ngraph/python/CMakeLists.txt index 85254c66a50573..39081891c4691c 100644 --- a/ngraph/python/CMakeLists.txt +++ b/ngraph/python/CMakeLists.txt @@ -18,28 +18,67 @@ cmake_minimum_required (VERSION 3.13) project (pyngraph) -include(ExternalProject) - -ExternalProject_Add( - pybind11 - GIT_REPOSITORY "https://github.com/pybind/pybind11.git" - GIT_TAG "v2.5.0" - SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/pybind11" - CONFIGURE_COMMAND "" - BUILD_COMMAND "" - INSTALL_COMMAND "" -) +if(NOT DEFINED OpenVINO_MAIN_SOURCE_DIR) + find_package(InferenceEngineDeveloperPackage) + find_package(ngraph REQUIRED) +endif() + +if(ngraph_FOUND) + message("ngraph version = {${ngraph_VERSION}}") +endif() -set(BUILD_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/build_wheel.py.in") -set(BUILD_PY "${CMAKE_CURRENT_BINARY_DIR}/build_wheel.py") -configure_file(${BUILD_PY_IN} ${BUILD_PY} @ONLY) +include(FetchContent) -add_custom_command( - DEPENDS pybind11 - OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/dist/ - POST_BUILD - WORKING_DIR ${CMAKE_CURRENT_BINARY_DIR} - COMMAND python ${BUILD_PY} +FetchContent_Declare( + pybind11 + GIT_REPOSITORY "https://github.com/pybind/pybind11.git" + GIT_TAG "v2.5.0" ) -add_custom_target(python_wheel DEPENDS ngraph ${CMAKE_CURRENT_BINARY_DIR}/dist/) +FetchContent_GetProperties(pybind11) +if(NOT pybind11_POPULATED) + FetchContent_Populate(pybind11) + add_subdirectory(${pybind11_SOURCE_DIR} ${pybind11_BINARY_DIR}) +endif() + +find_package(Python COMPONENTS Interpreter Development REQUIRED) + +if(PYTHON_FOUND) + set(PYTHON_VERSION python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}) + message("Python version={${Python_VERSION}}") +else() + message(FATAL_ERROR "Python was not found!") +endif() + +if(CMAKE_COMPILER_IS_GNUCXX) + add_compile_options(-fPIC) +endif() + +if (APPLE) + add_link_options(-stdlib=libc++) +endif() + +if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") + #disable warning: This operator was deprecated and will be removed with v0 operation. + add_compile_options(/wd4996) +endif() + +file(GLOB_RECURSE SOURCES src/pyngraph/*.cpp) +pybind11_add_module(_${PROJECT_NAME} MODULE ${SOURCES}) +target_include_directories(_${PROJECT_NAME} PRIVATE src) +target_link_libraries(_${PROJECT_NAME} PRIVATE ngraph::ngraph ngraph::onnx_importer) + +if(OpenVINO_MAIN_SOURCE_DIR OR InferenceEngineDeveloperPackage_FOUND) + ie_cpack_add_component(pyngraph_${PYTHON_VERSION}) + + install(TARGETS _${PROJECT_NAME} + DESTINATION python/${PYTHON_VERSION} + COMPONENT pyngraph_${PYTHON_VERSION}) + + install(DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/src/ngraph + DESTINATION python/${PYTHON_VERSION} + COMPONENT pyngraph_${PYTHON_VERSION} + USE_SOURCE_PERMISSIONS) + + ie_cpack(pyngraph_${PYTHON_VERSION}) +endif() diff --git a/ngraph/python/build_wheel.py.in b/ngraph/python/build_wheel.py.in deleted file mode 100644 index 974d4867c44538..00000000000000 --- a/ngraph/python/build_wheel.py.in +++ /dev/null @@ -1,81 +0,0 @@ -# ****************************************************************************** -# Copyright 2017-2020 Intel Corporation -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ****************************************************************************** - -import os -import subprocess -import sys -import venv - -print("Building ngraph wheel for Python {}".format(sys.version_info.major)) - -PYBIND_HEADERS_PATH = "@CMAKE_CURRENT_BINARY_DIR@/pybind11" -NGRAPH_CPP_BUILD_PATH = "@CMAKE_INSTALL_PREFIX@/@NGRAPH_COMPONENT_PREFIX@" -NGRAPH_ONNX_IMPORT_ENABLE = "@NGRAPH_ONNX_IMPORT_ENABLE@" -NGRAPH_VERSION = "@NGRAPH_WHEEL_VERSION@" -PYTHON_API_SOURCE_DIR = "@CMAKE_CURRENT_SOURCE_DIR@" -BUILD_DIR = "@CMAKE_CURRENT_BINARY_DIR@" - -BUILD_DEPS = ["setuptools", "wheel", "pip"] - -try: - venv_dir = os.path.join(os.path.curdir, "whl_build_venv") - print("Creating a virtualenv to build the wheel in: ", os.path.abspath(venv_dir)) - venv.create(venv_dir, with_pip=True) - - venv_python = ( - os.path.abspath(os.path.join(venv_dir, "Scripts", "python")) - if os.name == "nt" - else os.path.abspath(os.path.join(venv_dir, "bin", "python")) - ) - - print("Installing build dependencies...") - pip_install_cmd = [venv_python, "-m", "pip", "install", "-U"] - pip_install_cmd.extend(BUILD_DEPS) - subprocess.check_call(pip_install_cmd) - - build_env_variables = { - "PYBIND_HEADERS_PATH": PYBIND_HEADERS_PATH, - "NGRAPH_CPP_BUILD_PATH": NGRAPH_CPP_BUILD_PATH, - "NGRAPH_ONNX_IMPORT_ENABLE": NGRAPH_ONNX_IMPORT_ENABLE, - "NGRAPH_VERSION": NGRAPH_VERSION, - } - env = os.environ - env.update(build_env_variables) - - print("Running setup.py bdist_wheel") - build_log = subprocess.Popen( - [venv_python, os.path.join(PYTHON_API_SOURCE_DIR, "setup.py"), "bdist_wheel"], - stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, universal_newlines=True, - ) - - for line in build_log.stdout: - sys.stdout.write(line) - - print("Running setup.py sdist") - subprocess.check_call([venv_python, os.path.join(PYTHON_API_SOURCE_DIR, "setup.py"), "sdist"]) - - output_dir = os.path.join(PYTHON_API_SOURCE_DIR, "dist") - print("\n>>> NOTE: nGraph Python packages created in ", output_dir) - print("\n".join(os.listdir(output_dir))) - -except subprocess.CalledProcessError as err: - print("Could not complete the wheel building process") - print("Command that failed: ", err.cmd) - if err.stdout is not None: - print("Command std output: ", err.stdout.decode("utf-8")) - if err.stderr is not None: - print("Command err output: ", err.stderr.decode("utf-8")) - sys.exit(1) diff --git a/ngraph/python/setup.py b/ngraph/python/setup.py index 2ef9450443617b..94cf0c5a6369f6 100644 --- a/ngraph/python/setup.py +++ b/ngraph/python/setup.py @@ -14,214 +14,29 @@ # limitations under the License. # ****************************************************************************** -import distutils.ccompiler import os -import re +import pathlib +import shutil +import glob +import sysconfig import sys +import multiprocessing -import setuptools from setuptools import Extension, setup from setuptools.command.build_ext import build_ext +from setuptools.command.install_lib import install_lib +from setuptools.command.install import install as _install +from setuptools.command.develop import develop as _develop +from distutils.command.build import build as _build __version__ = os.environ.get("NGRAPH_VERSION", "0.0.0.dev0") PYNGRAPH_ROOT_DIR = os.path.abspath(os.path.dirname(__file__)) -PYNGRAPH_SRC_DIR = os.path.join(PYNGRAPH_ROOT_DIR, "src") -NGRAPH_DEFAULT_INSTALL_DIR = os.environ.get("HOME") -NGRAPH_PYTHON_DEBUG = os.environ.get("NGRAPH_PYTHON_DEBUG") +NGRAPH_ROOT_DIR = os.path.normpath(os.path.join(PYNGRAPH_ROOT_DIR, "..")) +OPENVINO_ROOT_DIR = os.path.normpath(os.path.join(PYNGRAPH_ROOT_DIR, "../..")) # Change current working dircectory to ngraph/python os.chdir(PYNGRAPH_ROOT_DIR) -debug_optimization_flags = [ - "O1", "O2", "O3", "O4", "Ofast", "Os", "Oz", "Og", "O", "DNDEBUG" -] - - -def find_ngraph_dist_dir(): - """Return location of compiled ngraph library home.""" - if os.environ.get("NGRAPH_CPP_BUILD_PATH"): - ngraph_dist_dir = os.environ.get("NGRAPH_CPP_BUILD_PATH") - else: - ngraph_dist_dir = os.path.join(NGRAPH_DEFAULT_INSTALL_DIR, "ngraph_dist") - - found = os.path.exists(os.path.join(ngraph_dist_dir, "include/ngraph")) - if not found: - print( - "Cannot find nGraph library in {} make sure that " - "NGRAPH_CPP_BUILD_PATH is set correctly".format(ngraph_dist_dir) - ) - sys.exit(1) - else: - print("nGraph library found in {}".format(ngraph_dist_dir)) - return ngraph_dist_dir - - -def find_pybind_headers_dir(): - """Return location of pybind11 headers.""" - if os.environ.get("PYBIND_HEADERS_PATH"): - pybind_headers_dir = os.environ.get("PYBIND_HEADERS_PATH") - else: - pybind_headers_dir = os.path.join(PYNGRAPH_ROOT_DIR, "pybind11") - - found = os.path.exists(os.path.join(pybind_headers_dir, "include/pybind11")) - if not found: - print( - "Cannot find pybind11 library in {} make sure that " - "PYBIND_HEADERS_PATH is set correctly".format(pybind_headers_dir) - ) - sys.exit(1) - else: - print("pybind11 library found in {}".format(pybind_headers_dir)) - return pybind_headers_dir - - -NGRAPH_CPP_DIST_DIR = find_ngraph_dist_dir() -PYBIND11_INCLUDE_DIR = find_pybind_headers_dir() + "/include" -NGRAPH_CPP_INCLUDE_DIR = NGRAPH_CPP_DIST_DIR + "/include" -if os.path.exists(os.path.join(NGRAPH_CPP_DIST_DIR, "lib")): - NGRAPH_CPP_LIBRARY_DIR = os.path.join(NGRAPH_CPP_DIST_DIR, "lib") -elif os.path.exists(os.path.join(NGRAPH_CPP_DIST_DIR, "lib64")): - NGRAPH_CPP_LIBRARY_DIR = os.path.join(NGRAPH_CPP_DIST_DIR, "lib64") -else: - print( - "Cannot find library directory in {}, make sure that nGraph is installed " - "correctly".format(NGRAPH_CPP_DIST_DIR) - ) - sys.exit(1) - -if sys.platform == "win32": - NGRAPH_CPP_DIST_DIR = os.path.normpath(NGRAPH_CPP_DIST_DIR) - PYBIND11_INCLUDE_DIR = os.path.normpath(PYBIND11_INCLUDE_DIR) - NGRAPH_CPP_INCLUDE_DIR = os.path.normpath(NGRAPH_CPP_INCLUDE_DIR) - NGRAPH_CPP_LIBRARY_DIR = os.path.normpath(NGRAPH_CPP_LIBRARY_DIR) - -NGRAPH_CPP_LIBRARY_NAME = "ngraph" -"""For some platforms OpenVINO adds 'd' suffix to library names in debug configuration""" -if len([fn for fn in os.listdir(NGRAPH_CPP_LIBRARY_DIR) if re.search("ngraphd", fn)]): - NGRAPH_CPP_LIBRARY_NAME = "ngraphd" - -ONNX_IMPORTER_CPP_LIBRARY_NAME = "onnx_importer" -if len([fn for fn in os.listdir(NGRAPH_CPP_LIBRARY_DIR) if re.search("onnx_importerd", fn)]): - ONNX_IMPORTER_CPP_LIBRARY_NAME = "onnx_importerd" - - -def _remove_compiler_flags(obj): - """Make pybind11 more verbose in debug builds.""" - for flag in debug_optimization_flags: - try: - if sys.platform == "win32": - obj.compiler.compile_options.remove("/{}".format(flag)) - else: - obj.compiler.compiler_so.remove("-{}".format(flag)) - obj.compiler.compiler.remove("-{}".format(flag)) - except (AttributeError, ValueError): - pass - - -def parallelCCompile( - self, - sources, - output_dir=None, - macros=None, - include_dirs=None, - debug=0, - extra_preargs=None, - extra_postargs=None, - depends=None, -): - """Build sources in parallel. - - Reference link: - http://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils - Monkey-patch for parallel compilation. - """ - # those lines are copied from distutils.ccompiler.CCompiler directly - macros, objects, extra_postargs, pp_opts, build = self._setup_compile( - output_dir, macros, include_dirs, sources, depends, extra_postargs - ) - cc_args = self._get_cc_args(pp_opts, debug, extra_preargs) - - # parallel code - import multiprocessing.pool - - def _single_compile(obj): - try: - src, ext = build[obj] - except KeyError: - return - self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) - - # convert to list, imap is evaluated on-demand - pool = multiprocessing.pool.ThreadPool() - list(pool.imap(_single_compile, objects)) - return objects - - -distutils.ccompiler.CCompiler.compile = parallelCCompile - - -def has_flag(compiler, flagname): - """Check whether a flag is supported by the specified compiler. - - As of Python 3.6, CCompiler has a `has_flag` method. - cf http://bugs.python.org/issue26689 - """ - import tempfile - - with tempfile.NamedTemporaryFile("w", suffix=".cpp") as f: - f.write("int main (int argc, char **argv) { return 0; }") - try: - compiler.compile([f.name], extra_postargs=[flagname]) - except setuptools.distutils.errors.CompileError: - return False - return True - - -def cpp_flag(compiler): - """Check and return the -std=c++11 compiler flag.""" - if sys.platform == "win32": - return "" # C++11 is on by default in MSVC - elif has_flag(compiler, "-std=c++11"): - return "-std=c++11" - else: - raise RuntimeError("Unsupported compiler -- C++11 support is needed!") - - -sources = [ - "pyngraph/axis_set.cpp", - "pyngraph/axis_vector.cpp", - "pyngraph/coordinate.cpp", - "pyngraph/coordinate_diff.cpp", - "pyngraph/dict_attribute_visitor.cpp", - "pyngraph/dimension.cpp", - "pyngraph/function.cpp", - "pyngraph/node.cpp", - "pyngraph/node_input.cpp", - "pyngraph/node_output.cpp", - "pyngraph/node_factory.cpp", - "pyngraph/ops/constant.cpp", - "pyngraph/ops/parameter.cpp", - "pyngraph/ops/result.cpp", - "pyngraph/ops/util/arithmetic_reduction.cpp", - "pyngraph/ops/util/binary_elementwise_arithmetic.cpp", - "pyngraph/ops/util/binary_elementwise_comparison.cpp", - "pyngraph/ops/util/binary_elementwise_logical.cpp", - "pyngraph/ops/util/index_reduction.cpp", - "pyngraph/ops/util/op_annotations.cpp", - "pyngraph/ops/util/regmodule_pyngraph_op_util.cpp", - "pyngraph/ops/util/unary_elementwise_arithmetic.cpp", - "pyngraph/passes/manager.cpp", - "pyngraph/passes/regmodule_pyngraph_passes.cpp", - "pyngraph/partial_shape.cpp", - "pyngraph/pyngraph.cpp", - "pyngraph/shape.cpp", - "pyngraph/strides.cpp", - "pyngraph/tensor_iterator_builder.cpp", - "pyngraph/types/element_type.cpp", - "pyngraph/types/regmodule_pyngraph_types.cpp", - "pyngraph/util.cpp", - "pyngraph/variant.cpp", - "pyngraph/rt_map.cpp", -] +NGRAPH_LIBS = ["ngraph", "onnx_importer"] packages = [ "ngraph", @@ -237,130 +52,171 @@ def cpp_flag(compiler): "ngraph.impl.passes", ] -sources = [PYNGRAPH_SRC_DIR + "/" + source for source in sources] - -include_dirs = [PYNGRAPH_SRC_DIR, NGRAPH_CPP_INCLUDE_DIR, PYBIND11_INCLUDE_DIR] +data_files = [] -library_dirs = [NGRAPH_CPP_LIBRARY_DIR] +with open(os.path.join(PYNGRAPH_ROOT_DIR, "requirements.txt")) as req: + requirements = req.read().splitlines() -libraries = [NGRAPH_CPP_LIBRARY_NAME, ONNX_IMPORTER_CPP_LIBRARY_NAME] +cmdclass = {} +for super_class in [_build, _install, _develop]: -extra_compile_args = [] -extra_link_args = [] + class command(super_class): + """Add user options for build, install and develop commands.""" -data_files = [ - ( - "lib", - [ - os.path.join(NGRAPH_CPP_LIBRARY_DIR, library) - for library in os.listdir(NGRAPH_CPP_LIBRARY_DIR) - if os.path.isfile(os.path.join(NGRAPH_CPP_LIBRARY_DIR, library)) - ], - ), -] + cmake_build_types = ["Release", "Debug", "RelWithDebInfo", "MinSizeRel"] + user_options = super_class.user_options + [ + ("config=", None, "Build configuration [{}].".format("|".join(cmake_build_types))), + ("jobs=", None, "Specifies the number of jobs to use with make."), + ("cmake-args=", None, "Additional options to be passed to CMake.") + ] -ext_modules = [ - Extension( - "_pyngraph", - sources=sources, - include_dirs=include_dirs, - define_macros=[("VERSION_INFO", __version__)], - library_dirs=library_dirs, - libraries=libraries, - extra_compile_args=extra_compile_args, - extra_link_args=extra_link_args, - language="c++", - ), -] + def initialize_options(self): + """Set default values for all the options that this command supports.""" + super().initialize_options() + self.config = None + self.jobs = None + self.cmake_args = None + cmdclass[super_class.__name__] = command -def add_platform_specific_link_args(link_args): - """Add linker flags specific for the OS detected during the build.""" - if sys.platform.startswith("linux"): - link_args += ["-Wl,-rpath,$ORIGIN/../.."] - link_args += ["-z", "noexecstack"] - link_args += ["-z", "relro"] - link_args += ["-z", "now"] - elif sys.platform == "darwin": - link_args += ["-Wl,-rpath,@loader_path/../.."] - link_args += ["-stdlib=libc++"] - elif sys.platform == "win32": - link_args += ["/LTCG"] - - -class BuildExt(build_ext): - """A custom build extension for adding compiler-specific options.""" - - def _add_extra_compile_arg(self, flag, compile_args): - """Return True if successfully added given flag to compiler args.""" - if has_flag(self.compiler, flag): - compile_args += [flag] - return True - return False - - def _add_debug_or_release_flags(self): - """Return compiler flags for Release and Debug build types.""" - if NGRAPH_PYTHON_DEBUG in ["TRUE", "ON", True]: - if sys.platform == "win32": - return ["/Od", "/Zi", "/RTC1"] - else: - return ["-O0", "-g"] - else: - if sys.platform == "win32": - return ["/O2"] - else: - return ["-O2", "-D_FORTIFY_SOURCE=2"] - def _add_win_compiler_flags(self, ext): - self._add_extra_compile_arg("/GL", ext.extra_compile_args) # Whole Program Optimization - self._add_extra_compile_arg("/analyze", ext.extra_compile_args) +class CMakeExtension(Extension): + """Build extension stub.""" - def _add_unix_compiler_flags(self, ext): - if not self._add_extra_compile_arg("-fstack-protector-strong", ext.extra_compile_args): - self._add_extra_compile_arg("-fstack-protector", ext.extra_compile_args) + def __init__(self, name, sources=None): + if sources is None: + sources = [] + super().__init__(name=name, sources=sources) - self._add_extra_compile_arg("-fvisibility=hidden", ext.extra_compile_args) - self._add_extra_compile_arg("-flto", ext.extra_compile_args) - self._add_extra_compile_arg("-fPIC", ext.extra_compile_args) - ext.extra_compile_args += ["-Wformat", "-Wformat-security"] +class BuildCMakeExt(build_ext): + """Builds module using cmake instead of the python setuptools implicit build.""" - def _customize_compiler_flags(self): - """Modify standard compiler flags.""" - try: - # -Wstrict-prototypes is not a valid option for c++ - self.compiler.compiler_so.remove("-Wstrict-prototypes") + cmake_build_types = ["Release", "Debug", "RelWithDebInfo", "MinSizeRel"] + user_options = [ + ("config=", None, "Build configuration [{}].".format("|".join(cmake_build_types))), + ("jobs=", None, "Specifies the number of jobs to use with make."), + ("cmake-args=", None, "Additional options to be passed to CMake.") + ] - except (AttributeError, ValueError): - pass + def initialize_options(self): + """Set default values for all the options that this command supports.""" + super().initialize_options() + self.build_base = "build" + self.config = None + self.jobs = None + self.cmake_args = None - def build_extensions(self): - """Build extension providing extra compiler flags.""" - self._customize_compiler_flags() + def finalize_options(self): + """Set final values for all the options that this command supports.""" + super().finalize_options() - for ext in self.extensions: - ext.extra_compile_args += [cpp_flag(self.compiler)] + for cmd in ["build", "install", "develop"]: + self.set_undefined_options(cmd, ("config", "config"), + ("jobs", "jobs"), + ("cmake_args", "cmake_args")) - if sys.platform == "win32": - self._add_win_compiler_flags(ext) + if not self.config: + if self.debug: + self.config = "Debug" else: - self._add_unix_compiler_flags(ext) - - add_platform_specific_link_args(ext.extra_link_args) - - ext.extra_compile_args += self._add_debug_or_release_flags() - - if sys.platform == "darwin": - ext.extra_compile_args += ["-stdlib=libc++"] - - if NGRAPH_PYTHON_DEBUG in ["TRUE", "ON", True]: - _remove_compiler_flags(self) - - build_ext.build_extensions(self) - - -with open(os.path.join(PYNGRAPH_ROOT_DIR, "requirements.txt")) as req: - requirements = req.read().splitlines() + self.announce("Set default value for CMAKE_BUILD_TYPE = Release.", level=4) + self.config = "Release" + else: + build_types = [item.lower() for item in self.cmake_build_types] + try: + i = build_types.index(str(self.config).lower()) + self.config = self.cmake_build_types[i] + self.debug = True if "Debug" == self.config else False + except ValueError: + self.announce("Unsupported CMAKE_BUILD_TYPE value: " + self.config, level=4) + self.announce("Supported values: {}".format(", ".join(self.cmake_build_types)), level=4) + sys.exit(1) + if self.jobs is None and os.getenv("MAX_JOBS") is not None: + self.jobs = os.getenv("MAX_JOBS") + self.jobs = multiprocessing.cpu_count() if self.jobs is None else int(self.jobs) + + def run(self): + """Run CMake build for modules.""" + for extension in self.extensions: + if extension.name == "_pyngraph": + self.build_cmake(extension) + + def build_cmake(self, extension: Extension): + """Cmake configure and build steps.""" + self.announce("Preparing the build environment", level=3) + plat_specifier = ".%s-%d.%d" % (self.plat_name, *sys.version_info[:2]) + self.build_temp = os.path.join(self.build_base, "temp" + plat_specifier, self.config) + build_dir = pathlib.Path(self.build_temp) + + extension_path = pathlib.Path(self.get_ext_fullpath(extension.name)) + + os.makedirs(build_dir, exist_ok=True) + os.makedirs(extension_path.parent.absolute(), exist_ok=True) + + # If ngraph_DIR is not set try to build from OpenVINO root + root_dir = OPENVINO_ROOT_DIR + bin_dir = os.path.join(OPENVINO_ROOT_DIR, "bin") + if os.environ.get("ngraph_DIR") is not None: + root_dir = PYNGRAPH_ROOT_DIR + bin_dir = build_dir + + self.announce("Configuring cmake project", level=3) + ext_args = self.cmake_args.split() if self.cmake_args else [] + self.spawn(["cmake", "-H" + root_dir, "-B" + self.build_temp, + "-DCMAKE_BUILD_TYPE={}".format(self.config), + "-DNGRAPH_PYTHON_BUILD_ENABLE=ON", + "-DNGRAPH_ONNX_IMPORT_ENABLE=ON"] + ext_args) + + self.announce("Building binaries", level=3) + + self.spawn(["cmake", "--build", self.build_temp, "--target", extension.name, + "--config", self.config, "-j", str(self.jobs)]) + + self.announce("Moving built python module to " + str(extension_path), level=3) + pyds = list(glob.iglob("{0}/**/{1}*{2}".format(bin_dir, + extension.name, + sysconfig.get_config_var("EXT_SUFFIX")), recursive=True)) + for name in pyds: + self.announce("copy " + os.path.join(name), level=3) + shutil.copy(name, extension_path) + + +class InstallCMakeLibs(install_lib): + """Finds and installs NGraph libraries to a package location.""" + + def run(self): + """Copy libraries from the bin directory and place them as appropriate.""" + self.announce("Adding library files", level=3) + + root_dir = os.path.join(OPENVINO_ROOT_DIR, "bin") + if os.environ.get("ngraph_DIR") is not None: + root_dir = pathlib.Path(os.environ["ngraph_DIR"]) / ".." + + lib_ext = "" + if "linux" in sys.platform: + lib_ext = ".so" + elif sys.platform == "darwin": + lib_ext = ".dylib" + elif sys.platform == "win32": + lib_ext = ".dll" + + libs = [] + for ngraph_lib in NGRAPH_LIBS: + libs.extend(list(glob.iglob("{0}/**/*{1}*{2}".format(root_dir, + ngraph_lib, lib_ext), recursive=True))) + if not libs: + raise Exception("NGraph libs not found.") + + self.announce("Adding library files" + str(libs), level=3) + + self.distribution.data_files.extend([("lib", [os.path.normpath(lib) for lib in libs])]) + self.distribution.run_command("install_data") + super().run() + + +cmdclass["build_ext"] = BuildCMakeExt +cmdclass["install_lib"] = InstallCMakeLibs setup( name="ngraph-core", @@ -369,12 +225,12 @@ def build_extensions(self): author="Intel Corporation", url="https://github.com/openvinotoolkit/openvino", license="License :: OSI Approved :: Apache Software License", - ext_modules=ext_modules, + ext_modules=[CMakeExtension(name="_pyngraph")], package_dir={"": "src"}, packages=packages, - cmdclass={"build_ext": BuildExt}, - data_files=data_files, install_requires=requirements, + data_files=data_files, zip_safe=False, extras_require={}, + cmdclass=cmdclass ) diff --git a/ngraph/python/tox.ini b/ngraph/python/tox.ini index d586faf7c31226..661517823d0f04 100644 --- a/ngraph/python/tox.ini +++ b/ngraph/python/tox.ini @@ -11,13 +11,9 @@ deps = flake8-bugbear pytest-xdist setenv = - NGRAPH_CPP_BUILD_PATH = {env:NGRAPH_CPP_BUILD_PATH:{homedir}/ngraph_dist} - NGRAPH_ONNX_IMPORT_ENABLE = {env:NGRAPH_ONNX_IMPORT_ENABLE:"FALSE"} - LD_LIBRARY_PATH = {env:LD_LIBRARY_PATH:{homedir}/ngraph_dist/lib} - DYLD_LIBRARY_PATH = {env:DYLD_LIBRARY_PATH:{homedir}/ngraph_dist/lib} - PYBIND_HEADERS_PATH = {env:PYBIND_HEADERS_PATH:} NGRAPH_BACKEND = {env:NGRAPH_BACKEND:"CPU"} PYTHONPATH = {env:PYTHONPATH} + ngraph_DIR = {env:NGRAPH_CPP_BUILD_PATH} passenv = http_proxy https_proxy