Bugfix #2389 develop flowchart (#2392)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,7 +1,7 @@
 ---
 name: Bug report
 about: Fix something that's not working
-title: ''
+title: 'Bugfix: '
 labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED PROJECT ASSIGNMENT, type: bug'
 assignees: ''
 

docs/Users_Guide/overview.rst

@@ -34,9 +34,9 @@ The MET code and documentation is maintained by the DTC in Boulder, Colorado. Th
 MET components
 ==============
 
-The major components of the MET package are represented in :numref:`overview-figure`. The main stages represented are input, reformatting, plotting, intermediate output, statistical analyses, and output and aggregation/analysis. The MET-TC package functions independently of the other MET modules, as indicated in the Figure. Each of these stages is described further in later sections. For example, the input and output formats are discussed in :numref:`data_io` as well as in the sections associated with each of the statistics modules. MET input files are represented on the far left.
+The major components of the MET package are represented in :numref:`overview-figure`. The main stages represented are input, reformatting, plotting, intermediate output, statistical analyses, and output and aggregation/analysis. Each of these stages is described further in later sections. For example, the input and output formats are discussed in :numref:`data_io` as well as in the sections associated with each of the statistics modules. MET input files are represented on the far left.
 
-The reformatting stage of MET consists of several tools which perform a variety of functions. The ASCII2NC, PB2NC, MADIS2NC, LIDAR2NC, and IODA2NC tools read a variety of point observation input file formats and, optionally, derive time summaries for each observing location. They all write to a common NetCDF point observation file format which can be read by the other MET tools. The Point2Grid tool reads that common NetCDF point observation file format and interpolates the point data onto a user-specified grid. The Regrid-Data-Plane, Shift-Data-Plane, MODIS-Regrid, and WWMCA-Regrid tools read a variety of gridded input file formats and interpolate user-requested input fields to a user-defined output grid. While the MET statistics tools can interpolate many input file formats in-memory and on-the-fly, manually regridding upstream is sometimes useful. The Pcp-Combine tool adds, subtracts, or derives fields across multiple time steps. It is often run to accumulate precipitation amounts into a user-specified time interval - if a user would like to verify over a different time interval than is included in their forecast or observational dataset. The Gen-Vx-Mask tool provides a variety of methods for creating bitmapped masking areas. Those masks can then be used to efficiently limit verification to the interior of a user-specified region in the downstream statistics tools. The Gen-Ens-Prod tool derives basic ensemble products (mean, spread, probabilities) from multiple gridded input ensemble members. The GSI tools reformat binary GSI diagnostic data to be read by the Stat-Analysis tool.
+The reformatting stage of MET consists of several tools which perform a variety of functions. The ASCII2NC, PB2NC, MADIS2NC, LIDAR2NC, and IODA2NC tools read a variety of point observation input file formats and, optionally, derive time summaries for each observing location. They all write to a common NetCDF point observation file format which can be read by the other MET tools. The Point2Grid tool reads that common NetCDF point observation file format or observations provided via Python and interpolates the point data onto a user-specified grid. The Regrid-Data-Plane, Shift-Data-Plane, MODIS-Regrid, and WWMCA-Regrid tools read a variety of gridded input file formats and interpolate user-requested input fields to a user-defined output grid. While the MET statistics tools can interpolate many input file formats in-memory and on-the-fly, manually regridding upstream is sometimes useful. The Pcp-Combine tool adds, subtracts, or derives fields across multiple time steps. It is often run to accumulate precipitation amounts into a user-specified time interval - if a user would like to verify over a different time interval than is included in their forecast or observational dataset. The Gen-Vx-Mask tool provides a variety of methods for creating bitmapped masking areas. Those masks can then be used to efficiently limit verification to the interior of a user-specified region in the downstream statistics tools. The Gen-Ens-Prod tool derives basic ensemble products (mean, spread, probabilities) from multiple gridded input ensemble members. The GSI tools reformat binary GSI diagnostic data to be read by the Stat-Analysis tool.
 
 .. _overview-figure:
 
@@ -46,13 +46,13 @@ The reformatting stage of MET consists of several tools which perform a variety
 
 Several optional plotting utilities are provided to assist users in checking their output from the data preprocessing step. Plot-Point-Obs creates a postscript plot showing the locations of point observations. This can be quite useful for assessing whether the latitude and longitude of observation stations was specified correctly. Plot-Data-Plane produces a similar plot for gridded data. For users of the MODE object based verification methods, the Plot-MODE-Field utility will create graphics of the MODE object output. Finally, WWMCA-Plot produces a plot of the raw WWMCA data file. 
 
-The main statistical analysis components of the current version of MET are: Point-Stat, Grid-Stat, Series-Analysis, Ensemble-Stat, MODE, MODE-TD (MTD), Grid-Diag, and Wavelet-Stat. The Point-Stat tool is used for grid-to-point verification, or verification of a gridded forecast field against a point-based observation (i.e., surface observing stations, ACARS, rawinsondes, and other observation types that could be described as a point observation). In addition to providing traditional forecast verification scores for both continuous and categorical variables, confidence intervals are also produced using parametric and non-parametric methods. Confidence intervals take into account the uncertainty associated with verification statistics due to sampling variability and limitations in sample size. These intervals provide more meaningful information about forecast performance. For example, confidence intervals allow credible comparisons of performance between two models when a limited number of model runs is available.
+The main statistical analysis components of the current version of MET are: Point-Stat, Grid-Stat, Series-Analysis, Ensemble-Stat, MODE, MODE-TD (MTD), Grid-Diag, and Wavelet-Stat. The Point-Stat tool is used for grid-to-point verification, or verification of a gridded forecast field against point observations (i.e., surface observing stations, ACARS, rawinsondes, and other observation types that could be described as a point observation). The point observations are read from the common NetCDF point observation file format or are supplied via Python. In addition to providing traditional forecast verification scores for both continuous and categorical variables, confidence intervals are also produced using parametric and non-parametric methods. Confidence intervals take into account the uncertainty associated with verification statistics due to sampling variability and limitations in sample size. These intervals provide more meaningful information about forecast performance. For example, confidence intervals allow credible comparisons of performance between two models when a limited number of model runs is available.
 
 Sometimes it may be useful to verify a forecast against gridded fields (e.g., Stage IV precipitation analyses). The Grid-Stat tool produces traditional verification statistics when a gridded field is used as the observational dataset. Like the Point-Stat tool, the Grid-Stat tool also produces confidence intervals. The Grid-Stat tool also includes "neighborhood" spatial methods, such as the Fractional Skill Score (:ref:`Roberts and Lean, 2008 <Roberts-2008>`). These methods are discussed in :ref:`Ebert (2008) <Ebert-2008>`. The Grid-Stat tool accumulates statistics over the entire domain.
 
 Users wishing to accumulate statistics over a time, height, or other series separately for each grid location should use the Series-Analysis tool. Series-Analysis can read any gridded matched pair data produced by the other MET tools and accumulate them, keeping each spatial location separate. Maps of these statistics can be useful for diagnosing spatial differences in forecast quality. 
 
-Ensemble-Stat is a hybrid tool that provides based post-processing capability of the ensemble members as well as computing measures of ensemble characteristics. Basic post-processing capability includes computing the ensemble mean, min, max, standard deviation, and ensemble relative frequency or probability. These fields can then be used in other MET tools for additional evaluation. Note however that the Gen-Ens-Prod tool also performs ensemble product generation, and this functionality will be removed from Ensemble-Stat in future versions. The ensemble characteristics include computation of rank and probability integral transform (PIT) histograms, the end-points for receiver operator curve (ROC) and reliability diagrams, and ranked probabilities scores (RPS) and the continuous version (CRPS).
+Ensemble-Stat compares ensemble member data to gridded analyses and/or point observations and computes measures of ensemble characteristics. The ensemble characteristics include ensemble mean and spread information, computation of rank and probability integral transform (PIT) histograms, the points for the receiver operator characteristic (ROC) and reliability diagrams, and ranked probabilities scores (RPS) and the continuous version (CRPS). When categorical thresholds are specified, Ensemble-Stat derives ensemble relative frequencies and verifies them as probability forecasts against the gridded analyses and/or point observations provided. Note that the ensemble post-processing provided in prior versions of this tool has moved to Gen-Ens-Prod.
 
 The MODE (Method for Object-based Diagnostic Evaluation) tool also uses gridded fields as observational datasets. However, unlike the Grid-Stat tool, which applies traditional forecast verification techniques, MODE applies the object-based spatial verification technique described in :ref:`Davis et al. (2006a,b) <Davis-2006>` and :ref:`Brown et al. (2007) <Brown-2007>`. This technique was developed in response to the "double penalty" problem in forecast verification. A forecast missed by even a small distance is effectively penalized twice by standard categorical verification scores: once for missing the event and a second time for producing a false alarm of the event elsewhere. As an alternative, MODE defines objects in both the forecast and observation fields. The objects in the forecast and observation fields are then matched and compared to one another. Applying this technique also provides diagnostic verification information that is difficult or even impossible to obtain using traditional verification measures. For example, the MODE tool can provide information about errors in location, size, and intensity.
 
@@ -62,11 +62,11 @@ The Grid-Diag tool produces multivariate probability density functions (PDFs) th
 
 The Wavelet-Stat tool decomposes two-dimensional forecasts and observations according to the Intensity-Scale verification technique described by :ref:`Casati et al. (2004) <Casati-2004>`. There are many types of spatial verification approaches and the Intensity-Scale technique belongs to the scale-decomposition (or scale-separation) verification approaches. The spatial scale components are obtained by applying a wavelet transformation to the forecast and observation fields. The resulting scale-decomposition measures error, bias and skill of the forecast on each spatial scale. Information is provided on the scale dependency of the error and skill, on the no-skill to skill transition scale, and on the ability of the forecast to reproduce the observed scale structure. The Wavelet-Stat tool is primarily used for precipitation fields. However, the tool can be applied to other variables, such as cloud fraction.
 
-Results from the statistical analysis stage are output in ASCII, NetCDF and Postscript formats. The Point-Stat, Grid-Stat, and Wavelet-Stat tools create STAT (statistics) files which are tabular ASCII files ending with a ".stat" suffix. In earlier versions of MET, this output format was called VSDB (Verification System DataBase). VSDB, which was developed by the NCEP, is a specialized ASCII format that can be easily read and used by graphics and analysis software. The STAT output format of the Point-Stat, Grid-Stat, and Wavelet-Stat tools is an extension of the VSDB format developed by NCEP. Additional columns of data and output line types have been added to store statistics not produced by the NCEP version.
+Results from the statistical analysis stage are output in ASCII, NetCDF and Postscript formats. The Point-Stat, Grid-Stat, Wavelet-Stat, and Ensemble-Stat tools create STAT (statistics) files which are tabular ASCII files ending with a ".stat" suffix. The STAT output files consist of multiple line types, each containing a different set of related statistics. The columns preceeding the LINE_TYPE column are common to all lines. However, the number and contents of the remaining columns vary by line type.
 
-The Stat-Analysis and MODE-Analysis tools aggregate the output statistics from the previous steps across multiple cases. The Stat-Analysis tool reads the STAT output of Point-Stat, Grid-Stat, Ensemble-Stat, and Wavelet-Stat and can be used to filter the STAT data and produce aggregated continuous and categorical statistics. The MODE-Analysis tool reads the ASCII output of the MODE tool and can be used to produce summary information about object location, size, and intensity (as well as other object characteristics) across one or more cases.
+The Stat-Analysis and MODE-Analysis tools aggregate the output statistics from the previous steps across multiple cases. The Stat-Analysis tool reads the STAT output of Point-Stat, Grid-Stat, Ensemble-Stat, and Wavelet-Stat and can be used to filter the STAT data and produce aggregated continuous and categorical statistics. Stat-Analysis also reads matched pair data (i.e. MPR line type) via python embedding. The MODE-Analysis tool reads the ASCII output of the MODE tool and can be used to produce summary information about object location, size, and intensity (as well as other object characteristics) across one or more cases.
 
-Tropical cyclone forecasts and observations are quite different than numerical model forecasts, and thus they have their own set of tools. The MET-TC package includes several modules: TC-Dland, TC-Pairs, TC-Stat, TC-Gen, TC-RMW, and RMW-Analysis. The TC-Dland module calculates the distance to land from all locations on a specified grid. This information can be used in later modules to eliminate tropical cyclones that are over land from being included in the statistics. TC-Pairs matches up tropical cyclone forecasts and observations and writes all output to a file. In TC-Stat, these forecast / observation pairs are analyzed according to user preference to produce statistics. TC-Gen evaluates the performance of Tropical Cyclone genesis forecast using contingency table counts and statistics. TC-RMW performs a coordinate transformation for gridded model or analysis fields centered on the current storm location. RMW-Analysis filters and aggregates the output of TC-RMW across multiple cases.
+Tropical cyclone forecasts and observations are quite different than numerical model forecasts, and thus they have their own set of tools. These consist of TC-Dland, TC-Pairs, TC-Stat, TC-Gen, TC-RMW, and RMW-Analysis. The TC-Dland module calculates the distance to land from all locations on a specified grid. This information can be used in later modules to eliminate tropical cyclones that are over land from being included in the statistics. TC-Pairs matches up tropical cyclone forecasts and observations and writes all output to a file. In TC-Stat, these forecast / observation pairs are analyzed according to user preference to produce statistics. TC-Gen evaluates the performance of Tropical Cyclone genesis forecast using contingency table counts and statistics. TC-RMW performs a coordinate transformation for gridded model or analysis fields centered on the current storm location. RMW-Analysis filters and aggregates the output of TC-RMW across multiple cases.
 
 The following sections of this MET User's Guide contain usage statements for each tool, which may be viewed if you type the name of the tool. Alternatively, the user can also type the name of the tool followed by **-help** to obtain the usage statement. Each tool also has a **-version** command line option associated with it so that the user can determine what version of the tool they are using.