Update the database schema to handle new columns and line types added to the MET-11.0.0-beta4 output.

[JohnHalleyGotway]

Describe the Enhancement

MET development version 11.0.0-beta4 made several changes to the "stat" output. This issue is to update the database schema accordingly to handle this new data for MET version 11.0.0.

• Issue Add the Mean Absolute Difference (SPREAD_MD) to the ECNT line type MET#2332 adds 1 new column (SPREAD_MD) to the ECNT line type generated by Ensemble-Stat and the HiRA method in Point-Stat. See Table 13.2. The SPREAD_MD is the ensemble mean absolute difference and is an alternative measure of ensemble spread. This column should be aggregated over multiple cases using a weighted average approach, where the weight is defined by the number of matched pairs in the TOTAL column.

• Issue Add MAE to the ECNT line type from Ensemble-Stat and for HiRA MET#2325 adds 2 new columns (MAE and MAE_OERR) to the ECNT line type generated by Ensemble-Stat and the HiRA method in Point-Stat. See Table 13.2. The MAE is the mean absolute error. These columns should be aggregated together over multiple cases using a weighted average approach, where the weight is defined by the number of matched pairs in the TOTAL column.

• Issue Add new bias ratio statistic to the ECNT line type from Ensemble-Stat and for HiRA MET#2058 adds 5 new columns (BIAS_RATIO, N_GE_OBS, ME_GE_OBS, N_LT_OBS, ME_LT_OBS) to the ECNT line type generated by Ensemble-Stat and the HiRA method in Point-Stat. See Table 13.2. All 5 columns are added so that the BIAS_RATIO can be aggregated over multiple runs, as described below.

  • N_GE_OBS is an integer and is aggregated over multiple runs by summing the values.
  • ME_GE_OBS is a float and is aggregated as a weighted average where the weights are defined by the N_GE_OBS column.
  • N_LT_OBS is an integer and is aggregated over multiple runs by summing the values.
  • ME_LT_OBS is a float and is aggregated as a weighted average where the weights are defined by the N_LT_OBS column.
  • BIAS_RATIO is a float >0. It aggregated by first aggregating ME_GE_OBS and ME_LT_OBS and then computing BIAS_RATIO = ME_GE_OBS / absolute value of ME_LT_OBS.

• Issue Enhance TC-Pairs to read hurricane model diagnostic files (e.g. SHIPS) and TC-Stat to filter the new data MET#392 adds a new TCDIAG line type to the .tcst output files generated by the TC-Pairs tool. The creation of this line type is optional and only occurs when the user supplies the -diag command line option to define the input TC diagnostics data. Each TCMPR line can be followed by at most one TCDIAG line to which it corresponds. The TCDIAG line type is defined in Table 24.3 of the User's Guide. Note that it is variable length and that length is determined by N_DIAG in column 17. For each diagnostic (1 .. N_DIAG), 2 columns are provided... a DIAG_i diagnostic name (string) and VALUE_i diagnostic value (floating point).

Note that for a given track, the diagnostics names (i.e. DIAG_i) will remain the same. However, its possible that different runs of TC-Pairs will write different diagnostics names to the output.

The purpose of these diagnostics is to be able to filter the TCMPR track data to which they correspond. The TC-Stat tool does that using the -diag_thresh and -init_diag_thresh job command options. For METdataio, if loading this data is obvious/straight-forward, please feel free to do so. If not, we can do that at a later time and just confirm that the loader skips these new lines.

Sample data for this issue can be found in the nightly build output in seneca:/d1/projects/MET/MET_regression/develop/NB20221103/MET-develop/test_output (or a more recent date).
And here's a tarfile of that output:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/NB20221103_develop_test_output.tar.gz

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
No sub-issues needed.

Relevant Deadlines

By the end of the beta4 development cycle.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @venitahagerty
• Select scientist(s) or no scientist required: none required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  Will create corresponding issues for aggregating/plotting this data.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues

Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version

• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[venitahagerty]

This issue was closed by completion of PR 139