Enhance the regrid dictionary to support the convert function and data censoring.

[JohnHalleyGotway]

Describe the Enhancement

The regrid dictionary configuration option in many of the MET tools enable the input data to be regridded to a user-specified grid. In addition, the censor_thresh/censor_val and convert configuration options may be specified to transform the data. These are applied immediately after reading the input gridded data but before any regridding is done.

This task it to also support these configuration options (censor_thresh/censor_val and convert) inside the regrid dictionary. So after applying the regridding logic, apply the user-specified censoring and conversion options.

regrid = {
   to_grid       = NONE;
   method        = NEAREST;
   width         = 1;
   vld_thresh    = 0.5;
   shape         = SQUARE;
   convert(x)    = x;
   censor_thresh = [];
   censor_val    = [];
}

Note, no need to add these to all of the existing default configuration file for the MET tools. However, be sure to update the data/config/README file with these details.

Also add corresponding command line options to the regrid_data_plane tool.

Time Estimate

2 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• No sub-issues required.

Relevant Deadlines

Needed for surrogate severe for HWT 20202.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required (John HG)
• Select scientist(s) or no scientist required (Tina Kalb and Lindsay Blank)

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones
• Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdb, METviewer, METexpress, METcalcpy, METplotpy
  No impact to other repos.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update unit tests.
  Ran full regression test on changes on dakota:

cd /d3/projects/MET/MET_test_branches/feature_1293_regrid
../MET/scripts/test_regression.sh develop-ref feature_1293_regrid \
>& test_regression_develop-ref_vs_feature_1293_regrid.log&

• Add/update documentation: Listed here during transition from Lyx to Sphinx:
  https://docs.google.com/document/d/1nCPBzu0eYognWWDHyocF3riuzydkF01N87cDWG3Bu0g
• Push local changes to GitHub.
• Submit a pull request to merge into develop, listing the <Issue Number> in the title.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

I am adding command line options to the regrid_data_plane tool, and I have a question about the option for convert(). Should we specify "convert(x) = 2*x;" on the command line as...
(1) -convert 2*x (and implicitly require that the equation be defined in terms of x)
(2) -convert_x 2*x (to make the command line option name indicate that it's defined in terms of x)
(3) -convert 'convert(a) = 2*a;' (so the convert string is a little config file, where you can name the variable whatever you'd like)

[jprestop]

I'm not sure if you only want feedback from Lindsay and Tina or from everyone... Personally, I like option 3 the best. I think it is the most clear.

[JohnHalleyGotway]

The regression test ran successfully with the only difference being one new output file from unit_regrid.xml:
regrid/regrid_data_plane_GFS_TO_G212_CONVERT_CENSOR.nc

And here's a plot of regridded data that has been converted from Kelvin to Celcius and censored so that values less than 0 are bad data. This demonstrates that both operations are working and in the correct order... convert first and censor second.

[JohnHalleyGotway]

On 4/10/2020, Tina Kalb tested Ensemble-Stat from branch feature_1293_regrid and confirmed that these changes enabled her new use of Ensemble-Stat to work as expected.

[CPKalb]

I tested this, and it worked as expected. GitHub forced me to add a .txt extension to be able to upload the config file. The config file can also be found on dakota:/home/kalb/surrogate_severe/estat_test/EnsembleStatConfig_ss.

Command:
/d3/projects/MET/MET_test_branches/feature_1293_regrid/MET-
feature_1293_regrid/met/bin/ensemble_stat 1 /d3/personal/kalb/model_applications/convection_allowing_models/surrogate_severe_calc/20190518/hrrr_ncep_2019051800f036.nc EnsembleStatConfig_ss
EnsembleStatConfig_ss.txt

[CPKalb]

Output image from ensemble_stat showing that the missing data is now 0!