PointStat: Probability of precipitation (pop) verification Brier score & ROC calculation

[engrmahadi]

I'd like to use Metplus for assessing rainfall probability. In the forecast file, there's a 3-hourly forecast with the APCP variable indicating the probability of rainfall exceeding 0.2m, ranging from 0 to 1 with intervals of 0.1. The observation file contains 3-hourly gauge rainfall data. My aim is to compute the Brier score and Brier skill score, incorporating an ROC curve. I've encountered difficulties running PointStat and additional information would be greatly appreciated. I've included sample data and a plot for reference. Eventually, I aim to generate plots similar to the attached one, varying rainfall thresholds.

Thank you.
Regards
Hasan
Performance Assessment Team
Bureau of Meteorology
Config File:
IMPROVER_PointStat.txt

[engrmahadi]

Forecast Netcdf:

[engrmahadi]

Example observed gauge rainfall file in csv format ( as netcdf file not supported here converted using ascii2nc tool)
20230303_03H_obs_aws_3h_rainfall.txt

20230303_00H_obs_aws_3h_rainfall.txt

[JohnHalleyGotway]

Hello @engrmahadi. I see you have some questions about using the MET Point-Stat tool to verify the probability of 3-hour precip exceeding 0.2 mm versus the 3-hourly gauge accumulations. And I'll presume that those 3-hour intervals are defined the same way between the model and gauge data.

I see that you've sent us your METplus configuration file named IMPROVER_PointStat.txt, along with some sample gauge data, and plots to illustrate what the input data and desired outputs look like.

You describe having "encountered difficulties running PointStat", and I'm not sure exactly what those are. So two tips...

• In the future, it would also be useful to send us the METplus log file to help debug any error messages or unexpected behavior you note there.
• You can attach NetCDF file to GitHub discussions. You just need to zip them up first. Or if the files are big you post them to our anonymous ftp site.

For now, I'll review the Point-Stat config file you sent and offer some advice:

• First, start by running Plot-Data-Plane to confirm that MET is able to read the forecast data:

plot_data_plane reprojected_IMPR_AUST_precip_accum-PT03H_prob_20230301_000000_threshold_0.2mm_003.nc plot.ps 'name="APCP"; level="(0,*,*)";'

And then look at the resulting image plot.ps to confirm that the values and placement on the earth look good.

• Second, you do NOT want to use prob_pct_thresh. That's used by the Ensemble-Stat tool. Instead please try defining the thresholds like this:

POINT_STAT_ONCE_PER_FIELD = True
#FCST_POINT_STAT_IS_PROB
FCST_IS_PROB = True
FCST_VAR1_NAME = APCP
FCST_VAR1_LEVELS = "(0,*,*)"
FCST_VAR1_THRESH = ==0.1

OBS_IS_PROB = False
OBS_VAR1_NAME = 3h_accum_precip
OBS_VAR1_LEVELS = Z0
OBS_VAR1_THRESH = >=0.2

The following don't apply to Point-Stat and can be removed:

#FCST_POINT_STAT_PROB_THRESH = ==0.2
#FCST_VAR1_OPTIONS = prob_pct_thresh = [ ==0.1];
#OBS_VAR1_OPTIONS = prob_pct_thresh = [ ==0.1];

• Third, the probability outputs from Point-Stat are contained in the PCT, PSTD, PJC, and PRC line types. So request those line types:

POINT_STAT_OUTPUT_FLAG_PCT = BOTH
POINT_STAT_OUTPUT_FLAG_PSTD = BOTH
POINT_STAT_OUTPUT_FLAG_PJC = BOTH
POINT_STAT_OUTPUT_FLAG_PRC = BOTH

Here BOTH could just be STAT but using BOTH also writes the output to a more human-readable format.

Please give these changes a try and let me know if you are able to make more progress and/or have more questions.

Thanks!