Enhance MODE CTS output file to include missing categorical statistics, including SEDI

[jhicks14]

Describe the Enhancement

Currently, the only forecast verification metrics within the MODE CTS text file includes: accuracy, HSS, CSI, HK, GSS, and bias. The METplus documentation shows that no other metrics can be added to the CTS text file. This enhancement would include adding SEDI (and, preferably, any metric the user would like to include via override) to the CTS text file in order to prevent the need to have a separate call to a tool like grid_stat.

Based on the discussion in the comments below, this issue is to enhance MODE's CTS output line type by adding the following 7 columns: LODDS, ORSS, EDS, SEDS, EDI, SEDI, BAGSS

These are the statistics included in the STAT CTS line type that are missing from the MODE CTS line type. We are not adding columns for HSS_EC and EC_VALUE since they are not needed by NOAA/CPC at this time and would require a change in the MODE config file to support it.

Time Estimate

The amount of work should be minimal considering there are already contingency table metrics, such as CSI, included within the CTS text file.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add SEDI (or at least the ability to add SEDI) to MODE cts output file
• Allow for user to add any contingency table metric to cts output file via an override call in the config file

Relevant Deadlines

MET release 12.0.0, ideally in the beta1 version

Funding Source

N/A

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
• Update METdataio to load the modified output from MODE
  • Update database to load 7 new MODE CTS statistics METdataio#226
• Update METviewer to confirm that all newly added CTS stats can be plotted. Per @TatianaBurek this is not needed since METviewer does not actually plot the existing MODE CTS stats either.
• Update METplus truth dataset after modifying output files.
  • Update Truth: Based on dtcenter/MET#2655 METplus#2327

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

@j-opatz and @jhicks14, I see this new issue added to MET to enhance the categorical statistics written by MODE.

I see that you're particularly interested in the SEDI statistic. I have always thought its a bit awkward that Point-Stat and Grid-Stat write the common CTS STAT line type, whereas MODE writes a more limited custom CTS line type. One main difference is the metadata contained in the header columns. The MODE metadata includes object definition information, such as the smoothing radius and object definition threshold.

I'd be inclined to use this issue to update MODE to call the write_cts_cols() function called by Point-Stat and Grid-Stat. We'd leave the metadata in the header columns unchanged, but update the data columns in the MODE CTS output to match those written by Point-Stat and Grid-Stat.

That'd provide more output, but would not support the requested "configurability" of what user-requested stats. In addition, all the _BCL and _BCU columns would remain set to the default NA value since we wouldn't implement bootstrapping.

(As a side note, I wish we'd never added those because bootstrapping through time is much more useful than bootstrapping in space in a single run.)

What do you think of that approach?

[jhicks14]

@j-opatz and @jhicks14, I see this new issue added to MET to enhance the categorical statistics written by MODE.

I see that you're particularly interested in the SEDI statistic. I have always thought its a bit awkward that Point-Stat and Grid-Stat write the common CTS STAT line type, whereas MODE writes a more limited custom CTS line type. One main difference is the metadata contained in the header columns. The MODE metadata includes object definition information, such as the smoothing radius and object definition threshold.

I'd be inclined to use this issue to update MODE to call the write_cts_cols() function called by Point-Stat and Grid-Stat. We'd leave the metadata in the header columns unchanged, but update the data columns in the MODE CTS output to match those written by Point-Stat and Grid-Stat.

That'd provide more output, but would not support the requested "configurability" of what user-requested stats. In addition, all the _BCL and _BCU columns would remain set to the default NA value since we wouldn't implement bootstrapping.

(As a side note, I wish we'd never added those because bootstrapping through time is much more useful than bootstrapping in space in a single run.)

What do you think of that approach?

Sounds like a good compromise to me!

[JohnHalleyGotway]

@jhicks14 we discussed this during the MET development meeting this afternoon and also on Slack. We considered multiple implementation options...

1. Simply adding the missing column as requested.
2. Updating MODE to write both CTC and CTS output lines similar to the Point-Stat and Grid-Stat tools but with the MODE header columns.
3. Shoehorning the MODE CTC/CTS output into real .stat output files with .stat header columns that could be loaded into METviewer.

After discussing and considering impacts to the METplus-Analysis tools, we decided the most prudent approach would be option 1. Listed below are the statistics present in .stat CTS lines from Point-Stat and Grid-Stat that are missing from the MODE CTS output:
LODDS, ORSS, EDS, SEDS, EDI, SEDI, BAGSS, HSS_EC, EC_VALUE

I'm wondering about the last two (HSS_EC, EC_VALUE). Are they actually needed here? I ask because adding them would require adding a new MODE config option to set the EC_VALUE which impacts the METplus MODE wrapper. Would their addition be useful to you or not needed?

[jhicks14]

@jhicks14 we discussed this during the MET development meeting this afternoon and also on Slack. We considered multiple implementation options...

1. Simply adding the missing column as requested.
2. Updating MODE to write both CTC and CTS output lines similar to the Point-Stat and Grid-Stat tools but with the MODE header columns.
3. Shoehorning the MODE CTC/CTS output into real .stat output files with .stat header columns that could be loaded into METviewer.

After discussing and considering impacts to the METplus-Analysis tools, we decided the most prudent approach would be option 1. Listed below are the statistics present in .stat CTS lines from Point-Stat and Grid-Stat that are missing from the MODE CTS output: LODDS, ORSS, EDS, SEDS, EDI, SEDI, BAGSS, HSS_EC, EC_VALUE

I'm wondering about the last two (HSS_EC, EC_VALUE). Are they actually needed here? I ask because adding them would require adding a new MODE config option to set the EC_VALUE which impacts the METplus MODE wrapper. Would their addition be useful to you or not needed?

That implementation option sounds good to me!

I don't think we would need the HSS_EC/EC_VALUE stats at this time.