Skip to content

Commit

Permalink
update TVM EP docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Valery Chernov committed Feb 15, 2022
1 parent 21a0ad1 commit 5d28ed9
Showing 1 changed file with 35 additions and 45 deletions.
80 changes: 35 additions & 45 deletions docs/execution-providers/TVM-ExecutionProvider.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,58 +5,31 @@ parent: Execution Providers
nav_order: 8
---

# TVM (TVM) Execution Provider
# TVM Execution Provider
{: .no_toc }

TVM is an execution provider for ONNX Runtime that is built on top of Apache TVM. It enables ONNX Runtime users to leverage Apache TVM model optimizations.
The TVM EP is currently in "Preview". It's been tested to work on a limited set of ONNX models on Linux. Future enhancements will include support on Windows and MacOS.

__NOTE__: This TVM execution provider was developed as a _standalone_ TVM execution provider. The code repo has references to _stvm_ as the name for this EP.
TVM EP is currently in "Preview". It's been tested to work on a handful of models on Linux, but not on Windows or MacOS.

## Contents
{: .no_toc }

* TOC placeholder
{:toc}

## Build

To use the TVM EP in ONNX Runtime (ORT), users first need to build Apache TVM and ONNX Runtime.

Note: some python packages may need to be upgraded/downgraded because both TVM and ORT with the TVM EP use the Python API. Alternatively, use modify PYTHONPATH to solve these conflicts.

### Build and configure TVM
## Build ONNX Runtime with the TVM Execution Provider

Install the minimal pre-requisites on Ubuntu/Debian like linux operating systems:
```bash
apt-get install -y python3 python3-dev python3-pip python3-setuptools gcc libtinfo-dev zlib1g-dev build-essential cmake libedit-dev libxml2-dev llvm-12
pip3 install numpy decorator attrs
```

Clone this [repo](https://github.com/microsoft/onnxruntime) using the `--recursive` flag to pull all associated dependencies


Build TVM from the tvm_update folder:

```bash
cd onnxruntime/cmake/external/tvm_update/
mkdir build
cd ./build
cmake -DCMAKE_BUILD_TYPE=Release -DUSE_LLVM=ON -DUSE_OPENMP=gnu -DUSE_MICRO=ON (If your machine is CUDA enabled -DUSE_CUDA=ON) ..
make -j <number of threads in build machine>
```

Set the environment variable PYTHONPATH to tell python where to find the TVM library:

```bash
export TVM_HOME=<path_to_onnx_runtime>/cmake/external/tvm_update
export PYTHONPATH=$TVM_HOME/python:${PYTHONPATH}
```

For more details on installing Apache TVM click [here](https://tvm.apache.org/docs/install/from_source.html)

### Build ONNX Runtime with the TVM Execution Provider
Also, the current implementation has `NVidia GPU` support for TVM EP. For now, you can use only `NVidia GPU` with CUDA Toolkit support.
To do this, make sure you have installed the NVidia driver and CUDA Toolkit.
More detailed instructions can be found on the [official page](https://developer.nvidia.com/cuda-toolkit).

Clone this repo.
In order to build ONNXRT you will need to have CMake 3.18 or higher. In Ubuntu 20.04 you can use the following commands to install the latest version of CMake:

```bash
Expand All @@ -74,22 +47,39 @@ sudo apt-get install kitware-archive-keyring
sudo apt-get install cmake
```

Build ONNX Runtime:
Build ONNX Runtime (TVM x86):
```bash
./build.sh --config Release --enable_pybind --build_wheel --parallel --skip_tests --skip_onnx_tests --use_tvm
```

Build ONNX Runtime (TVM with CUDA support):
```bash
./build.sh --config Release --enable_pybind --build_wheel --parallel --skip_tests --skip_onnx_tests --use_tvm --tvm_cuda_runtime
```

This command builds both `TVM` and `onnxruntime-tvm`. It creates two wheel, one for each project.
Build the python API for ONNX Runtime instead of using the standard package. Instructions for this are given below.

Package for TVM:
```bash
./build.sh --config Release --enable_pybind --build_wheel --skip_tests --parallel --use_stvm --skip_onnx_tests
cd <path_to_onnx_runtime>
python3 -m pip uninstall tvm -y
whl_path=$(find ./build/<OS_NAME>/Release/_deps/tvm-src/python/dist -name "*.whl")
python3 -m pip install $whl_path
```

Build the python API for ONNX Runtime instead of using the standard package:
Package for TVM EP:
```bash
cd <path_to_onnx_runtime>
pip3 uninstall onnxruntime onnxruntime-stvm -y
whl_path=$(find ./build/Linux/Release/dist -name "*.whl")
python3 -m pip uninstall onnxruntime onnxruntime-tvm -y
whl_path=$(find ./build/<OS_NAME>/Release/dist -name "*.whl")
python3 -m pip install $whl_path
```
Alternatively, you can set PYTHONPATH to tell python where to find the ONNXRT library:

Alternatively, you can set `PYTHONPATH` to tell python where to find the ONNXRT library and the TVM library.
```bash
export ORT_PYTHON_HOME=<path_to_onnx_runtime>/build/Linux/Release
export PYTHONPATH=$ORT_PYTHON_HOME:${PYTHONPATH}
export PYTHONPATH=<path_to_onnx_runtime>/build/<OS_NAME>/Release:${PYTHONPATH}
export PYTHONPATH=<path_to_onnx_runtime>/build/<OS_NAME>/Release/_deps/tvm-src/python:${PYTHONPATH}
```

## Configuration options
Expand All @@ -102,7 +92,7 @@ po = [dict(target=client_target,
tuning_file_path=client_tuning_logfile,
input_names = input_names_str,
input_shapes = input_shapes_str)]
tvm_session = onnxruntime.InferenceSession(model_path, providers=["StvmExecutionProvider"], provider_options=po)
tvm_session = onnxruntime.InferenceSession(model_path, providers=["TvmExecutionProvider"], provider_options=po)
```
<br>

Expand Down Expand Up @@ -134,11 +124,11 @@ Using the TVM EP with TVM tuning logs also requires users to turn off ONNX Runti
so = onnxruntime.SessionOptions()
so.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL

tvm_session = onnxruntime.InferenceSession(model_path, sess_options=so, providers=["StvmExecutionProvider"], provider_options=po)
tvm_session = onnxruntime.InferenceSession(model_path, sess_options=so, providers=["TvmExecutionProvider"], provider_options=po)
```

## Samples
- [Sample notebook for ResNet50 inference with TVM EP](https://github.com/octoml/onnxruntime/blob/STVM_EP_PR/docs/python/inference/notebooks/onnxruntime-stvm-tutorial.ipynb)
- [Sample notebook for ResNet50 inference with TVM EP](https://github.com/microsoft/onnxruntime/blob/master/docs/python/inference/notebooks/onnxruntime-tvm-tutorial.ipynb)

## Known issues
- At this moment, the TVM EP has only been verified on UNIX/Linux systems.
Expand Down

0 comments on commit 5d28ed9

Please sign in to comment.