Feature 1546 CI testing

[georgemccabe]

Expected Differences

• Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
  
  This PR implements automation via GitHub Actions to run unit tests and compare the results to the latest truth or reference output.

• Push events on feature/bugfix branches will trigger the compilation of the MET tools to ensure changes did not break them.
• Pull requests into develop or main_vX.Y branches will trigger all of the unit tests and comparing the results to truth data.
• Push events on branches that end with -ref triggers all of the unit tests and an automatic update of the truth data set on DockerHub so subsequent tests will use the new results as truth.
• Keywords in the last commit message before a push event can override some of these settings and trigger different events. For example, adding "ci-run-unit" in a commit message will run the unit tests and diff logic on a normal push event.

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [Yes]
  

@JohnHalleyGotway spent some time rearranging unit test XML files to improve the efficiency of the automated runs. Each unit test group now writes to an output directory that matches the name of the test. Some test files that take a long time have been split up into multiple test files so they can be run in parallel. Others have been split up so that they can be run directly after other test files that produce output that is needed as input for the tests. More work can be done to improve the speed of the unit tests. See the comments in the corresponding GitHub issue #1546 for more details.

Pull Request Testing

• Describe testing already performed for these changes:
  

Triggered many GitHub Actions workflows to ensure that the expected behavior occurs.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

• Review the event rules (on keyword in .github/workflows/unit_test.yml) and the job control rules (in .github/jobs/set_job_control.sh) to ensure that the expected behavior will run based on what triggered the workflow.
• Review the commit keywords to see if any other desired overrides should be provided
• Review the Actions log output for the run triggered by this pull request to ensure it looks as expected
• After this pull request is merged, review the log for the workflow triggered by the push to the develop branch
• If the develop branch run looks good, create a pull request from develop into develop-ref (may have to resolve conflicts first, see https://metplus.readthedocs.io/en/develop/Contributors_Guide/add_use_case.html#create-a-pull-request-from-develop-into-develop-ref for tips). This PR can be merged without review since the rules for a PR into a -ref branch do not run the unit tests.
• Review the logs for workflow triggered on the push to the develop-ref branch to ensure that the truth data is updated properly.
• Trigger a workflow from the push on a new feature branch using the "ci-run-unit" keyword and ensure that the diff tests result in no differences.

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]

We will want to document how all of this new functionality works, but we should first determine if what is implemented is sufficient or if changes should be made. We should create a new GitHub issue for writing up documentation for this to add to the MET Contributor's Guide.

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [Yes]
  

• Changes to unit test structure will cause differences
• Changes to the unit tests to avoid writing output files that contain bad characters such as colon (:) that cause failures in creating GitHub Actions artifacts because they voilate NTFS file name rules
• Changes to recent pull request merges that have not yet been added to the develop-ref branch.

• Please complete this pull request review by 2022/01/28.
  

Pull Request Checklist

See the METplus Workflow for details.

• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Linked issues with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[georgemccabe]

It looks like the pull request check is not showing up on this pull request. This may be due to the fact that the develop branch does not yet have the workflow files so it doesn't include it in the list of checks for that branch. The documentation pull request check is shown, which was already in the develop branch at the time of this PR. The workflow that kicked off from the pull request is here: https://github.com/dtcenter/MET/actions/runs/1757053931
After this PR is merged, we could use the next pull request into develop to test that the pull request check is included.

[georgemccabe]

There were a few failures in the pull request unit test run due to changes that were merged in from develop. I made the updates to the feature branch, then closed and reopened this pull request to trigger the pull request workflow. Here is the new workflow: https://github.com/dtcenter/MET/actions/runs/1757642185

[georgemccabe]

In the MET engineering meeting today, @jprestop reminded me that the event triggering rules should be set so that changes to the documentation (met/docs/**) should not trigger the unit test workflow. We should add this to the event rules. We had discussed only triggering on changes under met/src, but there may be other files outside of that directory that may warrant a new run.

[JohnHalleyGotway]

I approve of these changes. Thanks so much @georgemccabe for leading the way on this!

I downloaded the logs.zip and diff.zip artifacts and inspected the differences flagged in logs/comp_dir.log. They fall into the following categories:

1. The truth data is "missing" 217 files while the new output data is missing "201". These changes are all expected.

• In 3 output filenames, colons are changed to underscores.
• 12 new output files are added to the python output directory for the new tests added by Feature 1844 python embedding #2010.
• 4 new instances of CONUS.nc appear in the ref_config output which were added to enable unit_ref_config.xml to be split out into multiple jobs which run concurrently.
• All remaining differences are due to the output directory name changing.

head -221 logs/comp_dir.log | tail -217 > missing_from_truth
head -423 logs/comp_dir.log  | tail -201 > missing_from_output
cat missing_from_truth | sed 's%stat_analysis_[a-z]s%stat_analysis%g' | sed 's%ref_config_lead_[0-9][0-9]%ref_config%g' | sed 's/_indy//g' | sed 's/climatology_[0-9].[0-9]deg/climatology/g' | sort > missing_from_truth_MOD
cat missing_from_output | sort > missing_from_output_MOD
diff missing_from_truth_MOD missing_from_output_MOD

And here's those diffs:

<     ascii2nc/qc_out_2012-09-07_00_00_00.GRM_P_FCST.nc 
---
>     ascii2nc/qc_out_2012-09-07_00:00:00.GRM_P_FCST.nc 
94,111c94,95
<     pcp_combine/wrfout_d01_2008-08-08_12_00_00_PLEV_APCP06.nc 
<     plot_data_plane/wrfout_d01_2008-08-08_12_00_00_PLEV_NC_PINTERP_PRES_SFC.ps 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V.stat 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ecnt.txt 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ens.nc 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_orank.nc 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_orank.txt 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_phist.txt 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_relp.txt 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_rhist.txt 
<     python/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V_ssvar.txt 
<     python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps 
<     python/pb2nc_TMP.nc 
<     python/point_stat_GRIB1_NAM_GDAS_WINDS_120000L_20120409_120000V.stat 
<     ref_config/gen_vx_mask/CONUS.nc 
<     ref_config/gen_vx_mask/CONUS.nc 
<     ref_config/gen_vx_mask/CONUS.nc 
<     ref_config/gen_vx_mask/CONUS.nc 
---
>     pcp_combine/wrfout_d01_2008-08-08_12:00:00_PLEV_APCP06.nc 
>     plot_data_plane/wrfout_d01_2008-08-08_12:00:00_PLEV_NC_PINTERP_PRES_SFC.ps 

2. There are 16 additional errors that look like this:

ERROR: NetCDF file size difference exceeds 1 %

The size of the NetCDF file created via GHA differs from kiowa or seneca, perhaps due to differing block sizes. But none of the data actually differed. So these diffs can be ignored.

3. 5 png images differed. I manually spot-checked them and saw no discernible difference in the plots. So these diffs can be ignored.

Once this PR has been merged into develop, recommend merging develop into develop-ref to regenerate the truth data used for future unit test difference comparisons.