Testing

Testing

Nightly Test Results

ESMF is tested on many platforms and compiler combinations on a nightly basis. Go here to view the latest test results: http://earthsystemmodeling.org/esmf-test-summary/

Adding a New Machine the Automated Testing

Visit the esmf-test-scripts README file for a description of how to configure a new machine and deploy the test system to collect results on a nightly basis.

Kinds of ESMF Tests

1. Library unit and system tests are included in the library source code itself and can be run through the ESMF build system after the library is built.
2. Installation checks are sanity checks on the command line applications installed with ESMF.
3. NUOPC prototypes are skeleton applications that exercise the NUOPC API and protocols.
4. Extended regridding tests check the quality of ESMF regridding capabilities using a large number of combinations of realistic grids under higher processor counts.
5. Bitwise reproducibility tests test whether changes to the framework are expected to change answers when used in applications. Currently this includes interpolation weights and indices.
6. Memory leak tests use a runtime memory analysis tool to detect memory leaks in the library.
7. Ad-hoc tests are written to study particular behaviors such as profiling a part of the system undergoing optimization. They are discarded after the implementation is complete.

Testing Levels

Testing levels provide a way to easily communicate how extensively a particular version of ESMF has been tested, and by implication, the expected degree of stability. In general, smaller testing levels rely on faster and more localized test procedures to maximize development productivity. Higher testing levels require broader, more comprehensive, and more expensive testing procedures in order to ensure maximum portability, stability, and consistency of the framework.

Level	Criteria	Method
1	PASS if local branch testing in the developer's environment using make check reports no failures. (Assumes unit and/or system tests have been implemented capturing the new features or fix.)	Manual execution by developer
2	PASS if multiple OS and compiler configurations pass level 1 testing on the feature branch or the develop branch.	Continuous Integration (CI) when pushed to branch
3	PASS if a broad range of compiler and MPI combinations pass: (1) library unit and system tests, (2) NUOPC prototypes, and (3) extended regridding tests.	Automated "nightly" regression test scripts on a range of HPC platforms.
4	PASS if successfully tested in production (or near-production) environment(s) with target application(s) and all bit-reproducibility tests pass.	Manual execution of application in production environment. (May require customer assistance.)

Test Coverage

Test Coverage is run automatically through CircleCI on approval by the user. After a commit, on CircleCI find the run-test-coverage job under the test-coverage-commit workflow and click on the ARTIFACTS tab. The test coverage html files are available from there in a zip file for local download, if needed. The html files are also automatically pushed to the esmf-test-artifacts repo.

• Html files are pushed to test_coverage in the esmf-test-artifacts repository.

API Changes

API Changes is run automatically through CircleCI on approval by the user. After a commit, on CircleCI find the run-api-change job under the api-change-commit workflow and click on the ARTIFACTS tab. The api change and diff files of two different tags are available from there in a zip file for local download, if needed. The files are also automatically pushed to the esmf-test-artifacts repo.

• Api changes and diff files are pushed to api_change in the esmf-test-artifacts repository.

Running Test Coverage and API Change Locally

The CircleCI job about uses a set of Docker images that include all the dependencies for building the docs. Pre-built images are available on the ESMF DockerHub. To run the test_coverage and api_change script locally, use the docker commands below. (Otherwise, you will need to set up all the doc generation dependencies locally, which is quite complex.) The steps below are based on the automated commands seen in the CircleCI config.yml configuration file.

# install docker if not available, or use docker02.cgd.ucar.edu

# start with an empty working directory
cd /path/to/working/dir

# get docker image recipe for building docs
git clone https://github.com/ESCOMP/ESCOMP-Containers.git

# build the image  (set ESMF_BRANCH to desired branch from which to run test coverage)
docker build -t "esmf/test-coverage:develop" --file "ESCOMP-Containers/ESMF/test_coverage/Dockerfile" --build-arg ESMF_BRANCH="develop" --build-arg DOCKER_NAMESPACE="esmf" --no-cache .

# copy the generated docs out of the container to a local directory
# /path/to/working/dir should be an absolute path on your local machine where the html files will be copied
docker run -v /path/to/working/dir:/host-machine esmf/test-coverage:develop cp -r /artifacts /host-machine


# Similarly for API changes 
# build the image  (set ESMF_BRANCH to desired branch from which to run api changes)
# API changes is used to see change between two different tags TAG1 by default is "develop" and TAG2 is passed as an argument
docker build -t "esmf/api-change:develop" --file "ESCOMP-Containers/ESMF/API_changes/Dockerfile" --build-arg ESMF_BRANCH="develop" --build-arg DOCKER_NAMESPACE="esmf" --build-arg TAG2="ESMF_8_0_1" --no-cache .

# copy the generated docs out of the container to a local directory
# /path/to/working/dir should be an absolute path on your local machine where the .out files will be copied
docker run -v /path/to/working/dir:/host-machine esmf/api-change:develop cp -r /artifacts /host-machine

ESMPy

ESMPy testing requires access to data files. These files can be downloaded either from the ESMF website using the make download command (wget is required), or from the esmf-test-data github lfs repository:

git clone [email protected]:esmf-org/esmf-test-data.git

Note that the large file support is a paid service through github, so this strategy should be reserved for testing environments that require batch job submission, because they require cloning the test data on the login node prior to submitting a job to install and test esmpy on the compute nodes. When using this method for accessing the test data, the following environment variable must be set to the grids directory of the clone to esmf-test-data:

export ESMPY_DATA_DIR=<clone-of-esmf-test-data-repo>/grids