Feature #2374 v11.0.0 (#2375)

Co-authored-by: jprestop <[email protected]>

docs/Users_Guide/appendixC.rst

@@ -382,6 +382,29 @@ For cost / loss ratio above the base rate, the ECLV is defined as:
 
 .. math:: \text{ECLV } = \frac{(cl \ast (h + f)) + m - b}{b \ast (cl - 1)}.
 
+Stable Equitable Error in Probability Space (SEEPS)
+---------------------------------------------------
+
+Included in SEEPS output :numref:`table_PS_format_info_SEEPS` and SEEPS_MPR output :numref:`table_PS_format_info_SEEPS_MPR`
+
+The SEEPS scoring matrix (equation 15 from :ref:`Rodwell et al, 2010 <Rodwell-2010>`) is:
+
+.. math:: \{S^{S}_{vf}\} = \frac{1}{2}
+          \begin{Bmatrix}
+             0 & \frac{1}{1-p_1} & \frac{1}{p_3} + \frac{1}{1-p_1}\\
+             \frac{1}{p_1} & 0 & \frac{1}{p_3}\\
+             \frac{1}{p_1} + \frac{1}{1-p_3} & \frac{1}{1-p_3} & 0
+          \end{Bmatrix}
+
+In addition, Rodwell et al (2011) note that SEEPS can be written as the mean of two 2-category scores that individually assess the dry/light and light/heavy thresholds (:ref:`Rodwell et al., 2011 <Rodwell-2011>`). Each of these scores is like 1 – HK, but written as:
+
+.. math:: \frac{n_{01}}{\text{Expected n}_{.1}} + \frac{n_{10}}{\text{Expected n}_{.0}}
+
+
+where the word expected refers to the mean value deduced from the climatology, rather than the sample mean.
+
+SEEPS scores are expected to lie between 0 and 1, with a perfect forecast having a value of 0. Individual values can be much larger than 1. Results can be presented as a skill score by using the value of 1 – SEEPS.
+
 MET verification measures for continuous variables
 ==================================================
 

docs/Users_Guide/grid-stat.rst

@@ -71,6 +71,15 @@ There are several ways to present the results of the neighborhood approaches, su
 
 The user must specify several parameters in the grid_stat configuration file to utilize the neighborhood approach, such as the interpolation method, size of the smoothing window, and required fraction of valid data points within the smoothing window. For FSS-specific results, the user must specify the size of the neighborhood window, the required fraction of valid data points within the window, and the fractional coverage threshold from which the contingency tables are defined. These parameters are described further in the practical information section below.
 
+.. _grid-stat_seeps:
+
+SEEPS scores
+------------
+
+The Stable Equitable Error in Probability Space (SEEPS) was devised for monitoring global deterministic forecasts of precipitation against the WMO gauge network (:ref:`Rodwell et al., 2010 <Rodwell-2010>`; :ref:`Haiden et al., 2012 <Haiden-2012>`) and is a multi-category score which uses a climatology to account for local variations in behavior. Please see Point-Stat documentation :numref:`PS_seeps` for more details.
+
+The capability to calculate the SEEPS has also been added to Grid-Stat. This follows the method described in :ref:`North et al, 2022 <North-2022>`, which uses the TRMM 3B42 v7 gridded satellite product for the climatological values and interpolates the forecast and observed products onto this grid for evaluation. A 24-hour TRMM climatology (valid at 00 UTC) constructed from data over the time period 1998-2015 is supplied with the release. Expansion of the capability to other fields will occur as well vetted examples and funding allow.
+
 Fourier Decomposition
 ---------------------
 

docs/Users_Guide/point-stat.rst

@@ -136,6 +136,17 @@ The HiRA framework provides a unique method for evaluating models in the neighbo
 
 Often, the neighborhood size is chosen so that multiple models to be compared have approximately the same horizontal resolution. Then, standard metrics for probabilistic forecasts, such as Brier Score, can be used to compare those forecasts. HiRA was developed using surface observation stations so the neighborhood lies completely within the horizontal plane. With any type of upper air observation, the vertical neighborhood must also be defined.
 
+.. _PS_seeps:
+
+SEEPS scores
+------------
+
+The Stable Equitable Error in Probability Space (SEEPS) was devised for monitoring global deterministic forecasts of precipitation against the WMO gauge network (:ref:`Rodwell et al., 2010 <Rodwell-2010>`; :ref:`Haiden et al., 2012 <Haiden-2012>`) and is a multi-category score which uses a climatology to account for local variations in behavior. Since the score uses probability space to define categories using the climatology, it can be aggregated over heterogeneous climate regions. Even though it was developed for use with precipitation forecasts, in principle it could be applied to any forecast parameter for which a sufficiently long time period of observations exists to create a suitable climatology. The computation of SEEPS for precipitation is only supported for now.
+
+For use with precipitation, three categories are used, named ‘dry’, ‘light’ and ‘heavy’. The ‘dry’ category is defined (using the WMO observing guidelines) with any accumulation (rounded to the nearest 0.1 millimeter) that is less than or equal to 0.2 mm.  The remaining precipitation is divided into ‘light’ and ‘heavy’ categories whose thresholds are with respect to a climatology and thus location specific. The light precipitation is defined to occur twice as often as heavy precipitation.
+
+When calculating a single SEEPS value over observing stations for a particular region, the scores should have a density weighting applied which accounts for uneven station distribution in the region of interest (see Section 9.1 in :ref:`Rodwell et al., 2010 <Rodwell-2010>`). This density weighting has not yet been implemented in MET. Global precipitation climatologies calculated from the WMO SYNOP records from 1980-2009 are supplied with the release. At the moment, a 24-hour climatology is available (valid at 00 UTC or 12 UTC), but in future a 6-hour climatology will become available.
+
 .. _PS_Statistical-measures:
 
 Statistical measures

docs/Users_Guide/refs.rst

@@ -7,14 +7,14 @@ References
 .. _Aberson-1998:
 
 | Aberson, S.D., 1998: Five-day Tropical cyclone track forecasts in the North
-|   Atlantic Basin. *Weather & Forecasting*,  13, 1005-1015.
+|   Atlantic Basin. *Weather and Forecasting*,  13, 1005-1015.
 |
 
 .. _Ahijevych-2009:
 
-| Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009. Application of
+| Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009: Application of
 |   spatial verification methods to idealized and NWP-gridded precipitation forecasts.
-|   Weather Forecast., 24 (6), 1485 - 1497, doi: 10.1175/2009WAF2222298.1.
+|   *Weather and Forecasting*, 24 (6), 1485 - 1497, doi: 10.1175/2009WAF2222298.1.
 |
 
 .. _Barker-1991:
@@ -127,31 +127,30 @@ References
 
 .. _Gilleland-2017:
 
-| Gilleland, E., 2017. A new characterization in the spatial verification
+| Gilleland, E., 2017: A new characterization in the spatial verification
 |   framework for false alarms, misses, and overall patterns.
-|   Weather Forecast., 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.
+|   *Weather and Forecasting*, 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.
 |
 
-
 .. _Gilleland_PartI-2020:
 
-| Gilleland, E., 2020. Bootstrap methods for statistical inference.
+| Gilleland, E., 2020: Bootstrap methods for statistical inference.
 |   Part I: Comparative forecast verification for continuous variables.
-|   Journal of Atmospheric and Oceanic Technology, 37 (11), 2117 - 2134,
+|   *Journal of Atmospheric and Oceanic Technology*, 37 (11), 2117 - 2134,
 |   doi: 10.1175/JTECH-D-20-0069.1.
 |
 
 .. _Gilleland_PartII-2020:
 
-| Gilleland, E., 2020. Bootstrap methods for statistical inference.
-|   Part II: Extreme-value analysis. Journal of Atmospheric and Oceanic
-|   Technology, 37 (11), 2135 - 2144, doi: 10.1175/JTECH-D-20-0070.1.
+| Gilleland, E., 2020: Bootstrap methods for statistical inference.
+|   Part II: Extreme-value analysis. *Journal of Atmospheric and Oceanic*
+|   *Technology*, 37 (11), 2135 - 2144, doi: 10.1175/JTECH-D-20-0070.1.
 |
 
 .. _Gilleland-2021:
 
-| Gilleland, E., 2021. Novel measures for summarizing high-resolution forecast
-|   performance. Advances in Statistical Climatology, Meteorology and Oceanography,
+| Gilleland, E., 2021: Novel measures for summarizing high-resolution forecast
+|   performance. *Advances in Statistical Climatology, Meteorology and Oceanography*,
 |   7 (1), 13 - 34, doi: 10.5194/ascmo-7-13-2021.
 |
 
@@ -164,6 +163,14 @@ References
 |   http://www.stat.washington.edu/www/research/reports/
 | 
 
+.. _Haiden-2012:
+
+| Haiden, T., M.J. Rodwell, D.S. Richardson, A. Okagaki, T. Robinson, T. Hewson, 2012:
+|   Intercomparison of Global Model Precipitation Forecast Skill in 2010/11
+|   Using the SEEPS Score. *Monthly Weather Review*, 140, 2720-2733.
+|   https://doi.org/10.1175/MWR-D-11-00301.1
+|
+
 .. _Hamill-2001:
 
 | Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble
@@ -186,7 +193,7 @@ References
 
 | Knaff, J.A., M. DeMaria, C.R. Sampson, and J.M. Gross, 2003: Statistical,
 |   Five-Day Tropical Cyclone Intensity Forecasts Derived from Climatology
-|   and Persistence. *Weather & Forecasting,* Vol. 18 Issue 2, p. 80-92.
+|   and Persistence. *Weather and Forecasting,* Vol. 18 Issue 2, p. 80-92.
 | 
 
 .. _Mason-2004:
@@ -228,20 +235,44 @@ References
 |   verification. *Monthly Weather Review*, 115, 1330-1338.
 | 
 
+
+.. _North-2022:
+
+| North, R.C.,  M.P. Mittermaier, S.F. Milton, 2022. *Using SEEPS with a*
+|   TRMM-derived Climatology to Assess Global NWP Precipitation Forecast Skill.
+|   *Monthly Weather Review*, 150, 135-155.
+|   https://doi.org/10.1175/MWR-D-20-0347.1
+|
+
 .. _Ou-2016:
 
 | Ou, M. H., Charles, M., & Collins, D. C. 2016: Sensitivity of calibrated week-2
 |   probabilistic forecast skill to reforecast sampling of the NCEP global
-|   ensemble forecast system. *Weather and Forecasting,* 31(4), 1093-1107.
+|   ensemble forecast system. *Weather and Forecasting*, 31(4), 1093-1107.
 |
 
 .. _Roberts-2008:
 
 | Roberts, N.M., and H.W. Lean, 2008: Scale-selective verification of rainfall
 |   accumulations from high-resolution forecasts of convective events.
-|   *Monthly Weather Review,* 136, 78-97.
+|   *Monthly Weather Review*, 136, 78-97.
 | 
 
+.. _Rodwell-2010:
+
+| Rodwell, M.J., D.S. Richardson, T.D. Hewson and T. Haiden, 2010: A new equitable
+|   score suitable for verifying precipitation in numerical weather prediction.
+|   *Quarterly Journal of the Royal Meteorological Society*, 136: 1344-1463.
+|   https://doi.org/10.1002/qj.656
+|
+
+.. _Rodwell-2011:
+
+| Rodwell, M.J., T. Haiden, D.S. Richardson, 2011: Developments in Precipitation
+|   Verification. *ECMWF Newsletter* Number 128.
+|   https://www.ecmwf.int/node/14595
+|
+
 .. _Saetra-2004:
 
 | Saetra O., H. Hersbach, J-R Bidlot, D. Richardson, 2004: Effects of
@@ -279,22 +310,22 @@ References
 .. _Todter-2012:
 
 | Tödter, J. and B. Ahrens, 2012: Generalization of the Ignorance Score:
-|   Continuous ranked version and its decomposition. *Mon. Wea. Rev.*,
+|   Continuous ranked version and its decomposition. *Monthly Weather Review*,
 |   140 (6), 2005 - 2017, doi: 10.1175/MWR-D-11-00266.1.
 |
 
 .. _Weniger-2016:
 
 | Weniger, M., F. Kapp, and P. Friederichs, 2016: Spatial Verification Using
 |   Wavelet Transforms: A Review. *Quarterly Journal of the Royal*
-|   *Meteorological Society,* 143, 120-136.
+|   *Meteorological Society*, 143, 120-136.
 | 
 
 .. _Wilks-2010:
 
 | Wilks, D.S. 2010: Sampling distributions of the Brier score and Brier skill
 |   score under serial dependence. *Quarterly Journal of the Royal*
-|   *Meteorological Society,*, 136, 2109-2118. doi:10.1002/qj.709
+|   *Meteorological Society*, 136, 2109-2118. doi:10.1002/qj.709
 | 
 
 .. _Wilks-2011: