Initial Regression Test System

[ghost]

This giant pull request adds

• regression test framework
• regression test cases for openfast and beamdyn
• regression test documentation

The majority of the changes are confined to the new reg_tests directory. However, openfast/CMakeLists.txt was modified to add the BUILD_TESTING cmake option.

Each directory added contains a README.md with a description of its contents. The general instructions for testing are included in the sphinx documentation which can be built through the docs target. A quick start guide for running the regression test is found after building the documentation locally at
file://your/path/to/openfast/build/docs/html/source/test/regression_test.html#regression-test-from-scratch.

[michaelasprague]

@rafaelmudafort : please comment on the status of this testing framework for both double precision and single precision compilation of openfast.

[ghost]

@michaelasprague The regression test currently does not differentiate between single and double precision builds. Thus, running the regression test with a single precision executable against the double precision baseline solutions fails. Single precision tests could be supported by relaxing the tolerance in those tests.

However, I spent some time yesterday reevaluating the test pass/fail criteria and understanding what exactly is being compared (whats in the .outb, how is it being read and stored in the python regression test scripts, what does the tolerance on the norm mean). I'd like to characterize the regression test a bit more today and provide a summary later on.

[ghost]

@michaelasprague As mentioned above, the regression test system does not differentiate between single and double precision builds. However, it may not be appropriate to compare single precision solutions to those generated by a double precision executable due to differences that can arise in such complex nonlinear iterative system. To properly support both precisions in the regression test requires including and maintaining a solution set for each precision. I suggest making note of this limitation in the documentation instead.

In investigating the handling of single vs double precision in the regression test, the value that is currently set for the pass/fail tolerance came into question. The tolerance is a normalized magnitude of the difference between locally generated solutions and a designated baseline solution. It ought to take a value that is relevant to the question being asked when running the regression test. For example, if a user wants to verify that openfast was built correctly using the machine/compiler combinations outlined in the documentation, the tolerance should be close to machine tolerance. On the other hand, performance improvements (like OpenMP implementations) may affect the solution outside of but within a practical and acceptable engineering tolerance. I've made the regression test tolerance a CTEST variable in the CMake config so that users may now accept the default or determine their own appropriate tolerance value. I also suggest determining some appropriate tolerance values for particular cases to include in the documentation.