Skip to content

Commit

Permalink
[RFC] Pytest environment improvements (apache#5421)
Browse files Browse the repository at this point in the history
* [RFC] Pass pytest options globally.

In many places having a global pytest flag is useful . For me with the
build and test of tvm , I would like to be able to globally pass in
pytest options as part of development flow or CI flows where one would
like to measure other things regularly that need measurements including
pytest coverage data that I would like to experiment with across the stack.

This has been achieved with an additional setup-pytest-env.sh file in
tests/scripts rather than putting in something in every single task test
script and something I would like to avoid.

This now means the -v option to pytest is superfluous. I did consider
having a pytest.ini file but that doesn't allow me to pass any old
environment variable in and this seems to be the compromise.

* Improve other use case documentation

* Rationalize pytest environment.

* Remove the setting from docker/with_same_user.
* Take the opportunity to migrate common PYTHONPATH and
TVM_PATH into the common environment setting.

* Fixup vta fsim

* Be more explicit with common PYTHONPATH

* Fix python path for task_python_vta_fsim.sh properly

* Fix nit in documentation.
  • Loading branch information
Ramana Radhakrishnan authored and trevor-m committed Jun 18, 2020
1 parent 7cc370f commit 66557e8
Show file tree
Hide file tree
Showing 13 changed files with 66 additions and 36 deletions.
1 change: 1 addition & 0 deletions docker/bash.sh
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ ${DOCKER_BINARY} run --rm --pid=host\
-e "CI_BUILD_GROUP=$(id -g -n)" \
-e "CI_BUILD_GID=$(id -g)" \
-e "PYTHONPATH=python:topi/python"\
-e "CI_PYTEST_ADD_OPTIONS=$CI_PYTEST_ADD_OPTIONS" \
${CUDA_ENV}\
${CI_DOCKER_EXTRA_PARAMS[@]} \
${DOCKER_IMAGE_NAME}\
Expand Down
1 change: 1 addition & 0 deletions docker/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ ${DOCKER_BINARY} run --rm --pid=host \
-e "CI_BUILD_UID=$(id -u)" \
-e "CI_BUILD_GROUP=$(id -g -n)" \
-e "CI_BUILD_GID=$(id -g)" \
-e "CI_PYTEST_ADD_OPTIONS=$CI_PYTEST_ADD_OPTIONS" \
${CUDA_ENV}\
${CI_DOCKER_EXTRA_PARAMS[@]} \
${DOCKER_IMG_NAME} \
Expand Down
1 change: 1 addition & 0 deletions docker/with_the_same_user
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ getent passwd "${CI_BUILD_UID}" || adduser --gid "${CI_BUILD_GID}" --uid "${CI_B
--gecos "${CI_BUILD_USER} (generated by with_the_same_user script)" \
--disabled-password --home "${CI_BUILD_HOME}" --quiet "${CI_BUILD_USER}"
usermod -a -G sudo "${CI_BUILD_USER}"
# This is a grotesque hack to get PYTEST_ADD_OPTS available to all task scripts.
echo "${CI_BUILD_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-nopasswd-sudo

if [[ ! -z $CUDA_VISIBLE_DEVICES ]]; then
Expand Down
3 changes: 3 additions & 0 deletions docs/contribute/pull_request.rst
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,6 @@ If you want to run a single test:
rm -rf python/tvm/*.pyc python/tvm/*/*.pyc python/tvm/*/*/*.pyc
TVM_FFI=ctypes python -m pytest -v tests/python/unittest/test_pass_storage_rewrite.py
# Additionally if you want to run a single test, for example test_all_elemwise inside a file.
TVM_FFI=ctypes python -m pytest -v -k "test_all_elemwise" tests/python/frontend/tflite/test_forward.py
23 changes: 23 additions & 0 deletions tests/scripts/setup-pytest-env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -u
set -e
export TVM_PATH=`pwd`
export PYTHONPATH=${TVM_PATH}/python:${TVM_PATH}/topi/python
export PYTEST_ADDOPTS="-v $CI_PYTEST_ADD_OPTIONS"
1 change: 1 addition & 0 deletions tests/scripts/task_python_docs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
set -e
set -u

source tests/scripts/setup-pytest-env.sh
# cleanup old states
rm -rf docs/_build
mkdir -p docs/_build/html
Expand Down
20 changes: 10 additions & 10 deletions tests/scripts/task_python_frontend.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
set -e
set -u

export PYTHONPATH=python:topi/python
source tests/scripts/setup-pytest-env.sh
# to avoid openblas threading error
export TVM_BIND_THREADS=0
export OMP_NUM_THREADS=1
Expand All @@ -30,28 +30,28 @@ find . -type f -path "*.pyc" | xargs rm -f
make cython3

echo "Running relay TFLite frontend test..."
python3 -m pytest -v tests/python/frontend/tflite
python3 -m pytest tests/python/frontend/tflite

echo "Running relay MXNet frontend test..."
python3 -m pytest -v tests/python/frontend/mxnet
python3 -m pytest tests/python/frontend/mxnet

echo "Running relay Keras frontend test..."
python3 -m pytest -v tests/python/frontend/keras
python3 -m pytest tests/python/frontend/keras

echo "Running relay ONNX frontend test..."
python3 -m pytest -v tests/python/frontend/onnx
python3 -m pytest tests/python/frontend/onnx

echo "Running relay CoreML frontend test..."
python3 -m pytest -v tests/python/frontend/coreml
python3 -m pytest tests/python/frontend/coreml

echo "Running relay Tensorflow frontend test..."
python3 -m pytest -v tests/python/frontend/tensorflow
python3 -m pytest tests/python/frontend/tensorflow

echo "Running relay caffe2 frontend test..."
python3 -m pytest -v tests/python/frontend/caffe2
python3 -m pytest tests/python/frontend/caffe2

echo "Running relay DarkNet frontend test..."
python3 -m pytest -v tests/python/frontend/darknet
python3 -m pytest tests/python/frontend/darknet

echo "Running relay PyTorch frontend test..."
python3 -m pytest -v tests/python/frontend/pytorch
python3 -m pytest tests/python/frontend/pytorch
21 changes: 11 additions & 10 deletions tests/scripts/task_python_integration.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
set -e
set -u

export PYTHONPATH=`pwd`/python:`pwd`/topi/python:`pwd`/apps/extension/python
source tests/scripts/setup-pytest-env.sh
export PYTHONPATH=${PYTHONPATH}:${TVM_PATH}/apps/extension/python
export LD_LIBRARY_PATH="build:${LD_LIBRARY_PATH:-}"
export TVM_BIND_THREADS=0
export TVM_NUM_THREADS=2
Expand All @@ -42,26 +43,26 @@ rm -rf lib
make
cd ../..

TVM_FFI=cython python3 -m pytest -v apps/extension/tests
TVM_FFI=ctypes python3 -m pytest -v apps/extension/tests
TVM_FFI=cython python3 -m pytest apps/extension/tests
TVM_FFI=ctypes python3 -m pytest apps/extension/tests

# Test dso plugin
cd apps/dso_plugin_module
rm -rf lib
make
cd ../..
TVM_FFI=cython python3 -m pytest -v apps/dso_plugin_module
TVM_FFI=ctypes python3 -m pytest -v apps/dso_plugin_module
TVM_FFI=cython python3 -m pytest apps/dso_plugin_module
TVM_FFI=ctypes python3 -m pytest apps/dso_plugin_module

# Do not enable TensorFlow op
# TVM_FFI=cython sh prepare_and_test_tfop_module.sh
# TVM_FFI=ctypes sh prepare_and_test_tfop_module.sh

TVM_FFI=ctypes python3 -m pytest -v tests/python/integration
TVM_FFI=ctypes python3 -m pytest -v tests/python/contrib
TVM_FFI=ctypes python3 -m pytest tests/python/integration
TVM_FFI=ctypes python3 -m pytest tests/python/contrib

TVM_FFI=ctypes python3 -m pytest -v tests/python/relay
TVM_FFI=ctypes python3 -m pytest tests/python/relay

# Do not enable OpenGL
# TVM_FFI=cython python -m pytest -v tests/webgl
# TVM_FFI=ctypes python3 -m pytest -v tests/webgl
# TVM_FFI=cython python -m pytest tests/webgl
# TVM_FFI=ctypes python3 -m pytest tests/webgl
4 changes: 2 additions & 2 deletions tests/scripts/task_python_nightly.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@
set -e
set -u

export PYTHONPATH=python:topi/python
source tests/scripts/setup-pytest-env.sh

# Rebuild cython
make cython3

# cleanup pycache
find . -type f -path "*.pyc" | xargs rm -f

python3 -m pytest -v topi/tests/python/nightly
python3 -m pytest topi/tests/python/nightly
5 changes: 2 additions & 3 deletions tests/scripts/task_python_topi.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,11 @@
set -e
set -u

export PYTHONPATH=python:topi/python

source tests/scripts/setup-pytest-env.sh
# Rebuild cython
make cython3

# cleanup pycache
find . -type f -path "*.pyc" | xargs rm -f

python3 -m pytest -v topi/tests/python
python3 -m pytest topi/tests/python
6 changes: 3 additions & 3 deletions tests/scripts/task_python_unittest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@
set -e
set -u

export PYTHONPATH=python:topi/python
source tests/scripts/setup-pytest-env.sh

# cleanup pycache
find . -type f -path "*.pyc" | xargs rm -f

TVM_FFI=ctypes python3 -m pytest -v tests/python/unittest
TVM_FFI=ctypes python3 -m pytest tests/python/unittest
make cython3
TVM_FFI=cython python3 -m pytest -v tests/python/unittest
TVM_FFI=cython python3 -m pytest tests/python/unittest
8 changes: 4 additions & 4 deletions tests/scripts/task_python_vta_fsim.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
set -e
set -u

export TVM_PATH=`pwd`
export PYTHONPATH=${TVM_PATH}/python:${TVM_PATH}/vta/python:${TVM_PATH}/topi/python
source tests/scripts/setup-pytest-env.sh
export PYTHONPATH=${PYTHONPATH}:${TVM_PATH}/vta/python
export VTA_HW_PATH=`pwd`/3rdparty/vta-hw

# cleanup pycache
Expand All @@ -36,8 +36,8 @@ cp ${VTA_HW_PATH}/config/fsim_sample.json ${VTA_HW_PATH}/config/vta_config.json

# Run unit tests in functional/fast simulator
echo "Running unittest in fsim..."
python3 -m pytest -v ${TVM_PATH}/vta/tests/python/unittest
python3 -m pytest ${TVM_PATH}/vta/tests/python/unittest

# Run unit tests in functional/fast simulator
echo "Running integration test in fsim..."
python3 -m pytest -v ${TVM_PATH}/vta/tests/python/integration
python3 -m pytest ${TVM_PATH}/vta/tests/python/integration
8 changes: 4 additions & 4 deletions tests/scripts/task_python_vta_tsim.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
set -e
set -u

export TVM_PATH=`pwd`
export PYTHONPATH=${TVM_PATH}/python:${TVM_PATH}/vta/python:${TVM_PATH}/topi/python
source tests/scripts/setup-pytest-env.sh
export PYTHONPATH=${PYTHONPATH}:${TVM_PATH}/vta/python
export VTA_HW_PATH=`pwd`/3rdparty/vta-hw

# cleanup pycache
Expand Down Expand Up @@ -51,11 +51,11 @@ make -C ${VTA_HW_PATH}/hardware/chisel USE_THREADS=0 lib

# Run unit tests in cycle accurate simulator
echo "Running unittest in tsim..."
python3 -m pytest -v ${TVM_PATH}/vta/tests/python/unittest
python3 -m pytest ${TVM_PATH}/vta/tests/python/unittest

# Run unit tests in cycle accurate simulator
echo "Running integration test in tsim..."
python3 -m pytest -v ${TVM_PATH}/vta/tests/python/integration
python3 -m pytest ${TVM_PATH}/vta/tests/python/integration

# Reset default fsim simulation
cp ${VTA_HW_PATH}/config/fsim_sample.json ${VTA_HW_PATH}/config/vta_config.json

0 comments on commit 66557e8

Please sign in to comment.