Revamp regression test infrastructure

[charleskawczynski]

Purpose

The purpose of this SDI is to improve TC's testing infrastructure (i.e., upgrade to how ClimaAtmos' testing infrastructure works).

Cost/Benefits/Risks

The cost is low: users will need to become accustomed to the new workflow, which is only slightly different.

The benefit is that all of our tests will become "first class". Right now, we have two types of tests: "vanilla" tests and "non-vanilla" tests. Our non-vanilla tests only test that the code runs, and no tests are performed regarding the actual output values. So, if someone makes a "refactoring" change that is not actually refactoring, our vanilla tests will catch behavioral changes (i.e., they'll fail) while our non-vanilla "tests" will silently pass.

This means that results for any experimental models, e.g., new closures, can silently change in any PR.

One of the main advantages of applying these changes is that we've already implemented this solution in ClimaAtmos--much of what we did there is portable and we can simply move things over to TC.

People and Personnel

(@ the personnel required/requested to implement the software solution)

• Lead: @charleskawczynski
• Collaborators: @yairchn
• Reviewers: @trontrytel, @ilopezgp, @haakon-e, @costachris

Components

The testing infrastructure

Inputs

NCDatasets

Results and deliverables

Regression tests that work like ClimaAtmos

Task Breakdown And Tentative Due Date

We'll need to:

• Add a job_id Replace suffix with job_id in CL args #1206
• Export all regression variables to a single NC file in a callback
• Separate our testing / plotting infrastructure Split regression variables from plotted variables in CI #106. We can simply strip out regression tests of the existing TC regression infrastructure to start with, but this will open the door to generalizing (and maybe outsourcing) the plotting infrastructure into a separate package, which would be nice.

Proposed Delivery Date

I don't know that I have a delivery date, but the whole issue could probably be resolved in ~ 1-2 weeks.