Neighborhood Statistics - GridStat

[spence98]

Hello MET development team!

I am trying to use METplus to run the GridStat tool for neighborhood verification of WRF forecasts, specifically calculating the fractions skill score. I've followed the METplus use cases (5.2.3.1 and 5.2.3.6) and have successfully run those. However, when I edit those configuration files to run my own data and add the flags to support neighborhood statistics, the output files it produces are only including headers, without any actual calculations. I've confirmed that it will work and compute other statistics (the standard ones used in the use cases).

Would you be able to help me address this issue? My guess is that I am somehow calling the flags incorrectly or maybe missing one, but I'm unsure.

I've included the configuration files I'm using, my data files, and a log file from when I try to run it for you!

Michelle
metplus_files.zip

[jprestop]

Hi Michelle.

I see in your grid_stat_gridobs.conf file you have the following set:

GRID_STAT_NEIGHBORHOOD_COV_THRESH = >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=0.6, >=0.7, >=0.8, >=0.9
GRID_STAT_NEIGHBORHOOD_SHAPE = CIRCLE
GRID_STAT_NEIGHBORHOOD_WIDTH = 3, 5

I see in your metplus.log.20211118184453.txt log file that only the NEIGHBORHOOD_SHAPE and NEIGHBORHOOD_WIDTH values seem to be recognized:

11/18 18:44:53.836 metplus INFO: NEIGHBORHOOD_COV_THRESH=
11/18 18:44:53.836 metplus INFO: NEIGHBORHOOD_SHAPE=CIRCLE
11/18 18:44:53.836 metplus INFO: NEIGHBORHOOD_WIDTH=3, 5

I tried a few different configurations for COV_THRESH, but couldn't get any to work. I will need to request some assistance from @georgemccabe .

[georgemccabe]

Hi @spence98,

I ran your use case with the files that Julie downloaded (thanks, @jprestop!) and noticed a few things in the log output.

1. The environment variable NEIGHBORHOOD_COV_THRESH is not actually used in the wrapped MET config file. This is an old variable that is still set in case users are using very old MET config files. While it should be set properly here (thanks for bringing this to our attention!), it is not causing any issues in your use case because the env var that is used is:

11/19 09:25:22.373 metplus INFO: METPLUS_NBRHD_COV_THRESH=cov_thresh = [>=0.1, >=0.2, >=0.3, >=0.
4, >=0.5, >=0.6, >=0.7, >=0.8, >=0.9];

2. One of the fields is not being read properly:

WARNING: read_data_plane() -> MXPHL_prob_995(,) not found in file: /d1/projects/METplus/discuss
ions/1280/output/forecast_output/pp/oct/surrogate_severe_00_regrid_control.nc

This looks like a slight typo in the field name for one of the variables. It should be set to:

FCST_VAR3_NAME = MXUPHL_prob_995

3. The forecast and observation times do not line up.

WARNING: process_scores() -> Forecast and observation valid times do not match 20141014_100000 != 20141116_233000 for MXUPHL_prob_95(,) versus binary_obs(,).

The data will still be processed, but it will likely not give you useful results. I would suggest trying to process data that are valid at the same time. Let us know if you are still unable to get useful neighborhood results and we can take another look.

[spence98]

Thank you George and Julie!

Thanks for the information about the COV_THRESH variable, I was a little confused by the documentation as to why that was needing to be set and where it was being used for calculating the FSS.

Also, thank you for bringing my typo to my attention! I went into my configuration file and fixed that problem. However, when I tried to rerun the commands, it is still not producing any output for the neighborhood methods. I'm not getting any actual ERROR messages and it is still producing files with the headers, but no statistics are being calculated.

In regards to the forecast and observation times not lining up, they actually do in reality. I am processing a 24 hour period of my WRF simulations against the same 24 hour period of LSRs, but in order to get the pcpcombine, ensemble stat, and regrid data plane tools (equivalent to the 5.2.3.9 use case) I had to change my post processed WRF files so their extensions ranged from 00-24 (as opposed to anything else). I am not entirely sure why I was unable to read in files that were outside of that range, I couldn't find answers to that anywhere. In addition to that, when I ran the configuration for those three tools, each output file ended up with different "valid" times, for which I also wasn't able to track down the workflow into why it output the files were named as such. I'm not really sure how the best way to explain the issues I had is, but I'll include a little more detailed explanation below, in case you're curious.

Example of my workflow:

I have 36 hour long simulations. When I tried edit the 5.2.3.9 use case configuration file to do the same processes but with my data it was unable to read in 36 different files. It would read my post-processed files (named "WRFPRS_d01_## where ## ranged from 00-36) from 00-24 and then read in files 11-12 again, as opposed to reading in files 25-36. So, since I'm on a time crunch and wasn't able to quickly figure that issue out, my advisor and I chose 24 hour windows to narrow down our input forecast data to a window we knew MET would be able to read in. I updated my LSRs to reflect only those within that 24 hour period and then tried the tools again. For instance, hours 12-36 of my simulation for LSRs that occurred from 12-12. However, I still couldn't get MET to read in any files named after 24, so I changed the file names for each case so they all ranged from 00-24. I did this after trying to set a custom loop list, but that also did not work (produced an error for reading in the input file stating it couldn't be read with the template {custom...}). Renaming the files got me past the pcpcombine step, but when the file is produced from the ensemble stat tool, each case had a different valid time (regardless of what the actual timing was of my cases). I am honestly not sure how it was determining what the "valid" time is. This is why my files look like they don't match the observation times, when in reality I am processing 24 hour forecast data that lines up with the timing of the observations.

Michelle

[JohnHalleyGotway]

@spence98, George asked me to take a look at this discussion. It sounds like you're interested in computing the neighborhood fractions skill score (FSS). You're definitely correct, the coverage threshold is not used in the computation of FSS. FSS is one of the columns in the NBRCNT line type.

You should make sure that the NBRCNT line type is enabled in the output_flag, which you've already done:

GRID_STAT_OUTPUT_FLAG_NBRCNT = BOTH

An NBRCNT line type is written to the output for each combination of variable, categorical threshold, neighborhood, and confidence interval alpha value. However, neighborhood statistics are only computed for scalar variables, not probabilistic ones. Looking in "grid_stat_gridobs.conf" I see this:

FCST_IS_PROB = true

And that is likely preventing neighborhood stats from being computed for these inputs. Here's the line of code where that condition is enforced in Grid-Stat:
https://github.com/dtcenter/MET/blob/d1d8afb7d3004aad13d11f32f7ba10b62a03478a/met/src/tools/core/grid_stat/grid_stat.cc#L1326

If you want to compute probabilistic statistics for this data, as well as neighborhood statistics, you can do it in separate calls to METplus. In the second, we would NOT use the "FCST_IS_PROB" option.

[JohnHalleyGotway]

@spence98 I just want to followup to see if you have any remaining questions. If not, you could select the "Mark as answer" button for one of the previous responses. Once a discussion has been marked as answered, we then "Lock conversation" on it to prevent future posts. That encourages users to create new discussions for new questions, and also makes it easier for us to keep track of open questions.

[spence98]

Hi John,

Apologies for the delay. Things got crazy finishing up my degree and preparing for my defense. Having the FCST_IS_PROB flag set to true was what was preventing my neighborhood calculations from happening, so once I changed that I was able to move forward. Thank you, and the rest of the team, for all of your help!!