[UnitTests] Automatic parametrization over targets, with explicit opt…

…-out (apache#8010)

* [UnitTests] Explicitly list tests that were enabled by TVM_TEST_TARGETS but were skipped

Previously, these were removed by a filter in
tvm.testing._get_targets(), and weren't listed at all.  With this
change, they are instead removed by pytest.skipif, and show up as
explicitly skipped tests in pytest's summary when using
tvm.testing.parametrize_targets.

* [UnitTests] Automatic parametrize_targets for tests that use (target,dev)

Should make it easier to convert tests from using
tvm.testing.enabled_targets to use pytest's parametrized tests
instead.

* [UnitTests] Added ability to explicitly exclude a target from a particular test

Uses tvm_exclude_targets variable, which can be set (1) in the
conftest.py to apply to a test directory, (2) in a test script to
apply to that module, or (3) on an individual test function to apply
to it.  The @tvm.testing.exclude_targets decorator is provided for
readability in case #3.

* [UnitTests] Refactored test_topi_relu.py to use pytest.mark.parametrize

* [UnitTests] Added tvm_known_failing_targets option for the unittests.

Intended to mark tests that fail for a particular target, and are
intended to be fixed in the future.  Typically, these would result
either from implementing a new test, or from an in-progress
implementation of a new target.

* [UnitTests] Known failing targets now marked with xfail instead of skipif

* [UnitTests] Removed tvm_excluded_targets and tvm_known_failing_targets

These were implemented to exclude or mark as failing an entire file or
directory of tests.  In
https://discuss.tvm.apache.org/t/rfc-parametrized-unit-tests/9946/4,
it was pointed out that the global variables would be vulnerable to
typos in the names, resulting in the option being silently ignored.
The decorators `@tvm.testing.exclude_targets` and
`@tvm.testing.known_failing_targets` do not have this failure mode,
and are the preferred version.

* [UnitTests] Added helper functions to tvm.testing.

- tvm.testing.parameter() defines a parameter that can be passed to
  tests.  Tests that accept more than one parameter are run for all
  combinations of parameter values.

- tvm.testing.parameters() defines multiple sets of parameter values.
  Tests that accept more than one parameter are run once for each set
  of parameter values.

- tvm.testing.fixture() is a decorator that defines setup code.  The
  `cache=True` argument can be passed to avoid repeating expensive
  setup across multiple tests.

* [UnitTests] Bugfix for auto parametrizing of "target"

Previously, if the @parametrize_targets were present, but had other
@pytest.mark.parametrize after it, "target" would get parametrized a
second time.  Now, it checks more than just the closest "parametrize"
marker.

* [UnitTests] Renamed "cache" argument of tvm.testing.fixture to "cache_return_value"

* [UnitTests] Minor updates to parametrized test implementation.

As recommended by @tkonolige:

- Avoid infinite loop if LLVM target isn't enabled

- Update documentation for preferred use cases of
  tvm.testing.parametrize_targets, and recommended alternatives.

* [UnitTests] Minor updates to parametrized test implementation

- Documentation, removed previous example usage of tvm.testing.parametrize_targets

* [UnitTests] Changed accidental use of pytest fixtures to a NameError.

- Previously, a fixture function defined in a module was accessible
  through the global scope, and the function definition is accessible
  if a test function uses that name but fails to declare the fixture
  as a parameter.  Now, it will result in a NameError instead.

* [UnitTests] More careful removal of fixture functions from module global scope.

- Initial implementation only checked hasattr(obj, "_pytestfixturefunction")
  before removing obj, which gave false positives for objects that implement
  __getattr__, such as caffe.layers.  Now, also check that the value
  contained is a FixtureFunctionMarker.

* [UnitTests] Copy cached values when using tvm.testing.fixture(cache_return_value=True)

To avoid unit tests being able to influence each other through a
shared cache, all cached fixtures are passed through copy.deepcopy
prior to use.

* [UnitTests] Added meta-tests for tvm.testing functionality

Co-authored-by: Eric Lunderberg <[email protected]>

conftest.py

@@ -14,15 +14,33 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-import tvm.testing
+import pytest
 from pytest import ExitCode
 
+import tvm
+import tvm.testing
+
 
 def pytest_configure(config):
     print("enabled targets:", "; ".join(map(lambda x: x[0], tvm.testing.enabled_targets())))
     print("pytest marker:", config.option.markexpr)
 
 
+@pytest.fixture
+def dev(target):
+    return tvm.device(target)
+
+
+def pytest_generate_tests(metafunc):
+    tvm.testing._auto_parametrize_target(metafunc)
+    tvm.testing._parametrize_correlated_parameters(metafunc)
+
+
+def pytest_collection_modifyitems(config, items):
+    tvm.testing._count_num_fixture_uses(items)
+    tvm.testing._remove_global_fixture_definitions(items)
+
+
 def pytest_sessionfinish(session, exitstatus):
     # Don't exit with an error if we select a subset of tests that doesn't
     # include anything