002_setup_tests.md

002 - Setup Tests

Created: 2021-06-14

Last-updated: 2021-06-14

Contents:

• Summary
  • Issue
  • Decision
  • Status
• Details
  • Assumptions
  • Constraints
  • Positions
  • Argument
  • Implications
• Related
  • Related decisions
  • Related requirements
  • Related artifacts
  • Related principles
• Notes

Summary

Issue

We need to set up tests for all modules including C++ and Python codes. Currently, only scikit-image API (skimage module) has unit test cases and automated testing setup and no automated tests are available for clara module.

Test folder layout, test data, types of tests, libraries to use for testing need to be determined and properly documented.

Decision

Test folder layout

Use Tests outside application code except skimage Python module.

Test data

Generates test images at runtime as possible. For system tests, find representative test images and upload them to a publicly available site. Download/use those images locally.

Types of tests

• Unit tests
• Integration tests (integration with MONAI/Clara Train SDK)
• System tests
• Performance tests (load testing, checking overall memory usage)

Libraries to use

For C++:

Testing frameworks

• Use Catch2 over GoogleTest.

Benchmark

• Use GoogleBenchmark.

Dynamic Analysis

• Setup tests with Valgrind and Google Sanitizers.

For Python:

Testing frameworks

• Use PyTest.

Benchmark

• Use pytest-benchmark.

Profiler

• Provide a way to run a system test with Scalene.

Status

Decided. We are open to new alternatives as they arise.

Details

Assumptions

We want to set up automated testing on RAPIDS CI/CD infra.

We want to provide a way to execute small, medium, large/long-running tests.

We want to verify the integration with MONAI or Clara Train SDK.

We want to track the performance of loading/processing images.

We want to keep the current scikit-image API's test folder layout (tests folder beside each source module folder) and focus on folder layouts of Clara C++/Python modules.

Constraints

Downloading large test data is not available on CI/CD infra.

Positions

Test folder layout

We considered:

1. Tests outside application code
2. Tests as part of application code

Test data

We can choose one of the following approaches depending on the use case:

1. Generates test images at runtime
2. Uses representative test images downloaded

Types of tests

We can include the following types of tests:

• Unit tests
• Integration tests (integration with MONAI/Clara Train SDK)
• System tests
• Performance tests (load testing, checking overall memory usage)

Libraries to use

For C++:

Testing frameworks

1. GoogleTest
2. Catch2

Benchmark

• GoogleBenchmark

Dynamic Analysis

• Valgrind
• Google Sanitizers

For Python:

Testing frameworks

1. Built-in unittest package
2. PyTest
3. nose2

Benchmark

• pytest-benchmark

Profiler

• Scalene
  • https://youtu.be/nrQPqy3YY5A

Argument

Test folder layout

1. Tests outside application code: Fits well for most of our cases (except skimage package) because we also want to include other types of tests including integration/system tests.

2. Tests as part of application code: Good for unit tests and developers can easily correlate source code files with test files.

Test data

1. Generates test images at runtime: Good for small test cases (unit/integration tests) running at CI.

2. Uses representative test images downloaded: Good for long-running/system tests that require testing with realistic data.

Types of tests

• Unit tests: Can be run at CI.
• Integration tests (integration with MONAI/Clara Train SDK): Can be run at CI.
• System tests: We may want to provide a command to execute this test locally.
• Performance tests (load testing, checking overall memory usage): We may want to set up a different CI (NVIDIA internal infra) for this type of test.

Libraries to use

For C++:

Testing frameworks

1. GoogleTest: Popular framework. Supports multi-threads.
2. Catch2: No heavy dependency. Easy to implement test cases. See Why Catch.

Benchmark

• GoogleBenchmark: Popular framework. Provides good utility functions for the benchmark.

Dynamic Analysis

• Valgrind: We may want to run unit tests with valgrind.
  • Related to #38.
• Google Sanitizers
  • Related to #38.

For Python:

Testing frameworks (see https://www.softwaretestinghelp.com/python-testing-frameworks/)

1. Built-in unittest package: No dependency with 3rdparty package but test code is verbose compared with PyTest.
2. PyTest: Many functionalities and plugins. Easy to use. skimage module is already using this.
3. nose2: Extension of unittest.

Benchmark

• pytest-benchmark: This looks a good choice with PyTest (https://subscription.packtpub.com/book/application_development/9781787282896/1/ch01lvl1sec2/better-tests-and-benchmarks-with-pytest-benchmark).

Profiler

• Scalene: Looks better than existing profilers. See the PyCon 2021 video. We may want to use this locally during the development, instead of installing it at CI.
  • https://youtu.be/nrQPqy3YY5A

Implications

We may need to come up with a way to generate artificial test images and to find representative images for each image format.

Related

Related decisions

We will need to decide which system to use for measuring and tracking the performance.

Related requirements

The test cases implemented need to comply with Clara Train team's requirements (MONAI integration tests).

Related artifacts

We expect we may collect a set of test images and push it somewhere in public for local testing. We may want to create a test data repo in Github to store those images.

Related principles

Easy to implement test cases.

Notes