[CI] Run unit test with non-editable installation (#845)

We have been running unit test with editable installation. (i.e. `python setup.py develop`), with which we missed issues like #842. 

This CC makes installation in CI non-editable, and change test directory structure so that the source code will not shadow the installed version of `torchaudio`. With simple `pytest test`, `pytest` modifies `sys.path` and prepend checked out repository, which shadows the installed version.

To remedy this, the whole test suites has been moved from `./test` to `./test/torchaudio_unittest`. This adds nice module structure to our test code and we can do absolute import in each test module, which makes it possible again to run test with `python -m unittest torchaudio_unittest/XXX.py`

This change does not affect the regular development process (`python setup.py develop` && `pytest test`)

.circleci/unittest/linux/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-BUILD_SOX=1 python setup.py develop
+BUILD_SOX=1 python setup.py install

.circleci/unittest/linux/scripts/run_test.sh

@@ -15,8 +15,15 @@ python -m torch.utils.collect_env
 export TORCHAUDIO_TEST_FAIL_IF_NO_EXTENSION=1
 export PATH="${PWD}/third_party/install/bin/:${PATH}"
 
+declare -a common_args=(
+    '--cov=torchaudio'
+    "--junitxml=${PWD}/test-results/junit.xml"
+    '--durations' '20'
+)
 if [ "${os}" == MacOSX ] ; then
-    pytest -q -n auto --dist=loadscope --cov=torchaudio --junitxml=test-results/junit.xml --durations 20 test
+    declare -a args=('-q' '-n' 'auto' '--dist=loadscope')
 else
-    pytest -v --cov=torchaudio --junitxml=test-results/junit.xml --durations 20 test
+    declare -a args=('-v')
 fi
+cd test
+pytest "${args[@]}" "${common_args[@]}" torchaudio_unittest

.circleci/unittest/windows/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-python setup.py develop
+python setup.py install

.circleci/unittest/windows/scripts/run_test.sh

@@ -6,5 +6,5 @@ eval "$(./conda/Scripts/conda.exe 'shell.bash' 'hook')"
 conda activate ./env
 
 python -m torch.utils.collect_env
-pytest --cov=torchaudio --junitxml=test-results/junit.xml -v --durations 20 test
-flake8 torchaudio test
+cd test
+pytest --cov=torchaudio --junitxml=../test-results/junit.xml -v --durations 20 torchaudio_unittest

test/README.md

@@ -29,39 +29,39 @@ The following is an overview of the tests and related modules for `torchaudio`.
 ### Purpose specific test suites
 
 #### Numerical compatibility agains existing software
-- [Librosa compatibility test](./test_librosa_compatibility.py)
+- [Librosa compatibility test](./torchaudio_unittest/librosa_compatibility_test.py)
     Test suite for numerical compatibility against librosa.
-- [SoX compatibility test](./test_sox_compatibility.py)
+- [SoX compatibility test](./torchaudio_unittest/sox_compatibility_test.py)
     Test suite for numerical compatibility against SoX.
-- [Kaldi compatibility test](./test_kaldi_compatibility.py)
+- [Kaldi compatibility test](./torchaudio_unittest/kaldi_compatibility_test.py)
     Test suite for numerical compatibility against Kaldi.
 
 #### Result consistency with PyTorch framework
-- [TorchScript consistency test](torchscript_consistency_impl.py)
+- [TorchScript consistency test](./torchaudio_unittest/torchscript_consistency_impl.py)
     Test suite to check 1. if an API is TorchScript-able, and 2. the results from Python and Torchscript match.
-- [Batch consistency test](./test_batch_consistency.py)
+- [Batch consistency test](./torchaudio_unittest/batch_consistency_test.py)
     Test suite to check if functionals/Transforms handle single sample input and batch input and return the same result.
 
 ### Module specific test suites
 
 The following test modules are defined for corresponding `torchaudio` module/functions.
 
-- [`torchaudio.datasets`](./test_datasets)
-- `torchaudio.functional` [CPU](./functional_cpu_test.py), [CUDA](./functional_cuda_test.py), [common](./functional_impl.py)
-- [`torchaudio.transforms`](./test_transforms.py)
-- [`torchaudio.compliance.kaldi`](./test_compliance_kaldi.py)
-- [`torchaudio.kaldi_io`](./test_kaldi_io.py)
+- [`torchaudio.datasets`](./torchaudio_unittest/datasets)
+- [`torchaudio.functional`](./torchaudio_unittest/functional)
+- [`torchaudio.transforms`](./torchaudio_unittest/transforms_test.py)
+- [`torchaudio.compliance.kaldi`](./torchaudio_unittest/compliance_kaldi_test.py)
+- [`torchaudio.kaldi_io`](./torchaudio_unittest/kaldi_io_test.py)
 - [`torchaudio.sox_effects`](test/sox_effects)
-- [`torchaudio.save`, `torchaudio.load`, `torchaudio.info`](test/test_io.py)
+- [`torchaudio.save`, `torchaudio.load`, `torchaudio.info`](./torchaudio_unittest/io_test.py)
 
 ### Test modules that do not fall into the above categories
-- [test_dataloader.py](./test_dataloader.py)
+- [test_dataloader.py](./torchaudio_unittest/dataloader_test.py)
     Simple test for loading data and applying preprocessing.
 
 ### Support files
-- [assets](./assets): Contain sample audio files.
-- [assets/kaldi](./assets/kaldi): Contains Kaldi format matrix files used in [./test_compliance_kaldi.py](./test_compliance_kaldi.py).
-- [compliance](./compliance): Scripts used to generate above Kaldi matrix files.
+- [assets](./torchaudio_unittest/assets): Contain sample audio files.
+- [assets/kaldi](./torchaudio_unittest/assets/kaldi): Contains Kaldi format matrix files used in [./torchaudio_unittest/test_compliance_kaldi.py](./torchaudio_unittest/test_compliance_kaldi.py).
+- [compliance](./torchaudio_unittest/compliance): Scripts used to generate above Kaldi matrix files.
 
 ### Waveforms for Testing Purposes
 
@@ -116,13 +116,13 @@ Files:
 
 The following is the current practice of torchaudio test suite.
 
-1. Unless the tests are related to I/O, use synthetic data. [`common_utils`](./common_utils.py) has some data generator functions.
+1. Unless the tests are related to I/O, use synthetic data. [`common_utils`](./torchaudio_unittest/common_utils) has some data generator functions.
 1. When you add a new test case, use `common_utils.TorchaudioTestCase` as base class unless you are writing tests that are common to CPU / CUDA.
   - Set class memeber `dtype`, `device` and `backend` for the desired behavior.
   - If you do not set `backend` value in your test suite, then I/O functions will be unassigned and attempt to load/save file will fail.
   - For `backend` value, in addition to available backends, you can also provide the value "default" and backend will be picked automatically based on availability.
-1. If you are writing tests that should pass on diffrent dtype/devices, write a common class inheriting `common_utils.TestBaseMixin`, then inherit `common_utils.PytorchTestCase` and define class attributes (`dtype` / `device` / `backend`) there. See [Torchscript consistency test implementation](./torchscript_consistency_impl.py) and test definitions for [CPU](./torchscript_consistency_cpu_test.py) and [CUDA](./torchscript_consistency_cuda_test.py) devices.
-1. For numerically comparing Tensors, use `assertEqual` method from `common_utils.PytorchTestCase` class. This method has a better support for a wide variety of Tensor types.
+1. If you are writing tests that should pass on diffrent dtype/devices, write a common class inheriting `common_utils.TestBaseMixin`, then inherit `common_utils.PytorchTestCase` and define class attributes (`dtype` / `device` / `backend`) there. See [Torchscript consistency test implementation](./torchaudio_unittest/torchscript_consistency_impl.py) and test definitions for [CPU](./torchaudio_unittest/torchscript_consistency_cpu_test.py) and [CUDA](./torchaudio_unittest/torchscript_consistency_cuda_test.py) devices.
+1. For numerically comparing Tensors, use `assertEqual` method from torchaudio_unittest.common_utils.PytorchTestCase` class. This method has a better support for a wide variety of Tensor types.
 
 When you add a new feature(functional/transform), consider the following
 

test/test_backend.py → test/torchaudio_unittest/backend_test.py

@@ -1,6 +1,6 @@
 import torchaudio
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class BackendSwitchMixin:

test/test_batch_consistency.py → test/torchaudio_unittest/batch_consistency_test.py

@@ -7,7 +7,7 @@
 import torchaudio
 import torchaudio.functional as F
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class TestFunctional(common_utils.TorchaudioTestCase):

test/test_compliance_kaldi.py → test/torchaudio_unittest/compliance_kaldi_test.py

@@ -6,7 +6,7 @@
 import torchaudio
 import torchaudio.compliance.kaldi as kaldi
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 from .compliance import utils as compliance_utils
 
 

test/test_dataloader.py → test/torchaudio_unittest/dataloader_test.py

@@ -3,7 +3,7 @@
 import torchaudio
 from torch.utils.data import Dataset, DataLoader
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class TORCHAUDIODS(Dataset):

test/datasets/cmuarctic_test.py → test/torchaudio_unittest/datasets/cmuarctic_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets import cmuarctic
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/commonvoice_test.py → test/torchaudio_unittest/datasets/commonvoice_test.py

@@ -3,7 +3,7 @@
 import random
 
 from torchaudio.datasets import commonvoice
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/datasets_test.py → test/torchaudio_unittest/datasets/datasets_test.py

@@ -2,7 +2,7 @@
 from torchaudio.datasets.speechcommands import SPEECHCOMMANDS
 from torchaudio.datasets.vctk import VCTK
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TorchaudioTestCase,
     get_asset_path,
 )

test/datasets/gtzan_test.py → test/torchaudio_unittest/datasets/gtzan_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets import gtzan
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/librispeech_test.py → test/torchaudio_unittest/datasets/librispeech_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets import librispeech
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/libritts_test.py → test/torchaudio_unittest/datasets/libritts_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets.libritts import LIBRITTS
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/ljspeech_test.py → test/torchaudio_unittest/datasets/ljspeech_test.py

@@ -3,7 +3,7 @@
 
 from torchaudio.datasets import ljspeech
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/speechcommands_test.py → test/torchaudio_unittest/datasets/speechcommands_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets import speechcommands
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/datasets/utils_test.py → test/torchaudio_unittest/datasets/utils_test.py

@@ -4,7 +4,7 @@
 from torchaudio.datasets import utils as dataset_utils
 from torchaudio.datasets.commonvoice import COMMONVOICE
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_asset_path,

test/datasets/yesno_test.py → test/torchaudio_unittest/datasets/yesno_test.py

@@ -2,7 +2,7 @@
 
 from torchaudio.datasets import yesno
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     get_whitenoise,

test/functional_cpu_test.py → test/torchaudio_unittest/functional/functional_cpu_test.py

@@ -7,7 +7,7 @@
 from parameterized import parameterized
 import pytest
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 from .functional_impl import Lfilter
 
 

test/functional_cuda_test.py → test/torchaudio_unittest/functional/functional_cuda_test.py

@@ -1,6 +1,6 @@
 import torch
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 from .functional_impl import Lfilter
 
 

test/functional_impl.py → test/torchaudio_unittest/functional/functional_impl.py

@@ -2,7 +2,7 @@
 import torch
 import torchaudio.functional as F
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class Lfilter(common_utils.TestBaseMixin):

test/test_io.py → test/torchaudio_unittest/io_test.py

@@ -9,7 +9,7 @@
 from torchaudio.utils import sox_utils
 from torchaudio._internal.module_utils import is_module_available
 
-from .common_utils import get_asset_path
+from torchaudio_unittest.common_utils import get_asset_path
 
 BACKENDS = []
 BACKENDS_MP3 = []

test/kaldi_compatibility_cpu_test.py → test/torchaudio_unittest/kaldi_compatibility_cpu_test.py

@@ -1,6 +1,6 @@
 import torch
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 from .kaldi_compatibility_impl import Kaldi
 
 

test/kaldi_compatibility_cuda_test.py → test/torchaudio_unittest/kaldi_compatibility_cuda_test.py

@@ -1,6 +1,6 @@
 import torch
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 from .kaldi_compatibility_impl import Kaldi
 
 

test/kaldi_compatibility_impl.py → test/torchaudio_unittest/kaldi_compatibility_impl.py

@@ -7,7 +7,7 @@
 import torchaudio.compliance.kaldi
 from parameterized import parameterized
 
-from .common_utils import (
+from torchaudio_unittest.common_utils import (
     TestBaseMixin,
     load_params,
     skipIfNoExec,

test/test_kaldi_io.py → test/torchaudio_unittest/kaldi_io_test.py

@@ -3,7 +3,7 @@
 import torch
 import torchaudio.kaldi_io as kio
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class Test_KaldiIO(common_utils.TorchaudioTestCase):

test/test_librosa_compatibility.py → test/torchaudio_unittest/librosa_compatibility_test.py

@@ -17,7 +17,7 @@
 
 import pytest
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 @unittest.skipIf(not LIBROSA_AVAILABLE, "Librosa not available")

test/test_models.py → test/torchaudio_unittest/models_test.py

@@ -1,7 +1,7 @@
 import torch
 from torchaudio.models import Wav2Letter, MelResNet, UpsampleNetwork, WaveRNN
 
-from . import common_utils
+from torchaudio_unittest import common_utils
 
 
 class TestWav2Letter(common_utils.TorchaudioTestCase):

test/test_sox_compatibility.py → test/torchaudio_unittest/sox_compatibility_test.py

@@ -5,7 +5,7 @@
 import torchaudio.transforms as T
 from parameterized import parameterized
 
-from .common_utils import (
+from torchaudio_unittest.common_utils import (
     skipIfNoSoxBackend,
     skipIfNoExec,
     TempDirMixin,

test/sox_effect/common.py → test/torchaudio_unittest/sox_effect/common.py

@@ -2,7 +2,7 @@
 
 from parameterized import param
 
-from ..common_utils import get_asset_path
+from torchaudio_unittest.common_utils import get_asset_path
 
 
 def name_func(func, _, params):

test/sox_effect/test_dataset.py → test/torchaudio_unittest/sox_effect/dataset_test.py

@@ -4,7 +4,7 @@
 import torch
 import torchaudio
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     PytorchTestCase,
     skipIfNoExtension,

test/sox_effect/smoke_test.py → test/torchaudio_unittest/sox_effect/smoke_test.py

@@ -1,7 +1,7 @@
 from torchaudio import sox_effects
 from parameterized import parameterized
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     TorchaudioTestCase,
     skipIfNoExtension,

test/sox_effect/test_sox_effect.py → test/torchaudio_unittest/sox_effect/sox_effect_test.py

@@ -3,7 +3,7 @@
 from torchaudio import sox_effects
 from parameterized import parameterized
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     PytorchTestCase,
     skipIfNoExtension,

test/sox_effect/test_torchscript.py → test/torchaudio_unittest/sox_effect/torchscript_test.py

@@ -4,7 +4,7 @@
 from torchaudio import sox_effects
 from parameterized import parameterized
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     PytorchTestCase,
     skipIfNoExtension,

test/sox_io_backend/test_info.py → test/torchaudio_unittest/sox_io_backend/info_test.py

@@ -3,7 +3,7 @@
 
 from torchaudio.backend import sox_io_backend
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     PytorchTestCase,
     skipIfNoExec,

test/sox_io_backend/test_load.py → test/torchaudio_unittest/sox_io_backend/load_test.py

@@ -3,7 +3,7 @@
 from torchaudio.backend import sox_io_backend
 from parameterized import parameterized
 
-from ..common_utils import (
+from torchaudio_unittest.common_utils import (
     TempDirMixin,
     PytorchTestCase,
     skipIfNoExec,