Unit Testing

Unit testing has been added to EnergyPlus with tests located in the tst/EnergyPlus/unit subdirectory. A small set of "starter" tests has been committed to demonstrate some of the key usage.

Googletest (gtest)

We are using Googletest, also known as gtest. It offers a lot of nice capabilities and a low burden on developers to add new tests. Some of the features are:

• New tests are added by just putting the test file in the unit test directory (and adding the file to the CMakeLists.txt if building with CMake -- until we automate that)
• "Death" tests to check that exceptions or assertions should fire with certain inputs
• Tolerant equality checks for floating point values
• Simple setup/teardown support for tests with dependencies
• Filtering so that only certain tests are run (good while trying to correct test failures)
• Nice colored reports

Googletest documentation can be found here.

Writing Tests

Writing tests is simple:

• In CMake check the BUILD_TESTING option to add unit tests to the configuration
• Add a file (if it doesn't exist) with a name of the form _MyClass_.unit.cc or _MyFunction_.unit.cc
  • Copying an existing file is a quick way to get the boilerplate structure
• Add the file name to the test_src list in CMakeLists.txt in the unit directory
• Include the necessary headers for the code being tested
• Add using declarations to simplify the names you must type
• Add tests, where each test can include as few or as many checks as you want

The body of the simplest kind of test looks like this:

TEST( TestCaseName, TestName )
{
  ... // Test body goes here
}

Each test in a file should have the same ''TestCaseName'' and a different ''TestName''. CamelCase without underscores is suggested to avoid internal gtest problems at macro expansion time.

Tests may include some application object declarations, operations, and function calls interspersed with some gtest check statements, such as EXPECT_EQ( a, b );, which says that a and b should be equal. If the check fails then the test fails and gtest will report the test and line number where the failure occurred.

Running Tests

The tests will build into a single executable that runs all the tests in the same folder with other build targets. EnergyPlus_tests.exe reports test results to standard output.

To run a subset of the tests you can add a --gtest_filter argument such as:

EnergyPlus_tests.exe --gtest_filter="MyClassTest.*"

to run all tests within one test case. Other wildcard usage is supported to run, for example, a set of tests with similar test names.
When on Windows and running in Visual Studio debugger, the --gtest_filter can be passed as Command Argument so the debugger will startup at the desired unit test -- see Properties->Configuration Properties->Debugging-Command Arguments.

For example, you can run:

EnergyPlus_tests.exe --gtest_filter="EnergyPlusFixture.HeatBalanceKiva*"

To get the full list of available tests:

EnergyPlus_tests.exe --gtest_list_tests

Getting Fancier

There are a bunch of different checks beyond EXPECT_EQ. Some of other ones are EXPECT_NE, EXPECT_LT, EXPECT_GT, EXPECT_LE, EXPECT_GE, EXPECT_TRUE, and EXPECT_FALSE. There are checks that are tolerant for floating point values such as EXPECT_DOUBLE_EQ to allow for some roundoff and since precision can vary across platforms/compilers/builds.

To run tests on components that need some external state set up the tests use a class where the setup normally happens in the constructor and the teardown (to restore the prior state) happens in the destructor. The DataPlant.unit.cc file has an example of setup/teardown. Note that TEST_F is then used for the tests instead of TEST.

Strategy

Some strategies for growing the unit testing coverage of EnergyPlus:

• Tackle the easier code first:
  • Code with fewer dependencies, esp. on global data: setting up the state to run meaningful tests of functions with many dependencies is hard to do, which is why some codes use "mock" objects just for testing.
  • Simple functions with clearly defined inputs, outputs, and actions
• Skip hard-to-test functions that are expected to be refactored soon into OO code
• Add unit tests for any new classes and functions
• Most EnergyPlus classes are just containers at this point so there isn't much to test but as functions are brought into the classes tests should be added
• For code that is too hard to unit test currently at least use assertions to verify pre and post conditions
• As you work on code think about refactoring it to make it unit test friendly

clear_state()

As unit tests are added to new parts of the legacy code base, it will often be necessary to add, or expand existing, clear_state() routines. The purpose of clear_state is to return to the state the program would be in if the executable had completely exited and restarted. Include all non-static, non-const variables in the namespace. The variables should be set to the value they are assigned initially (often 0). All vectors/arrays should be deallocated. Basically you are returning all the variables to their original state as if the code is run for the first time.

Some one-time setup flags that are static in the routines may need to be hoisted to the namespace so they can be cleared. It is a good idea to prepend the name of the routine to the name of the one-time flag. You could put them in an unnamed namespace so that other files cannot access them. You don't have to move up all the static vars that are down in the routines, just those that control program flow and would otherwise keep things from being allocated again.

To test if your clear_state() is working it is a good idea to temporarily add an exact duplicate of a unit test to run right after the original and see if anything misbehaves. Once the duplicate runs fine, it is good to go. Not a guarantee that something else may still be lurking that may get tripped up in a future unit test, but at least you know everything triggered by the new unit test is being cleared.

EnergyPlusFixture and Severe and Fatal Errors

When using a GetInput function with EnergyPlusFixture and the process_idf() function sometimes input errors occur since only a few objects are usually sent to process_idf. These problems are usually reported in the eplusout.err file but that is not always created when unit testing. One approach is to see the tests in the console window is to include the following right after the GetInput calls

compare_err_stream( "" );

This will work unless there is a fatal error, in this case temporarily comment out the ShowFatalError call and you should be able to see the eplusout.err errors in the console window.

If you are testing something that should result in a warning or severe error and it is then calling the ShowFatalError near the end of the GetInput, you might want to follow the example in SetPointManager.cc. The original GetSetPointManageInput() routine was renamed GetSetPointManagerInputData(bool &ErrorsFound) and a new wrapper was added called GetSetPointMangerInput(). The ShowFatalError was moved to the wrapper. This allows the GetSetPointManagerInputData() to be called from a unit test without the ShowFatalError being called plus you can now test if the error was found or not. See the wrapper below:

void
GetSetPointManagerInputs()
{
   // wrapper for GetInput to allow unit testing when fatal inputs are detected
   static bool ErrorsFound( false );
   static std::string_view const RoutineName( "GetSetPointManagerInputs: " ); // include trailing blank space

   GetSetPointManagerInputData( ErrorsFound );

   if ( ErrorsFound ) {
      ShowFatalError( RoutineName + "Errors found in input.  Program terminates." );
   }
} // GetSetpointManagerInputs

Tips

• "Death" tests are nice but can slow down unit test runs on Windows (badly with MinGW GCC) and cause problems when multithreading test runs due to forking on Linux. Probably best to stay away from them or put them in a separate set of tests to be run manually, unthreaded only.
• On Windows, you will want to add the _NO_DEBUG_HEAP=1 Environment setting for the energyplus_tests project. This is because tests using EnergyPlusFixture read in the IDD and that is very slow unless this is set -- see Properties->Configuration Properties->Debugging->Environment .