Feature 1736 out_stat #1744
Merged
Feature 1736 out_stat #1744
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…, do not write output to the -out or log output.
…pulating the output AsciiTable unless it's actually going to be written.
…nto feature_1736_out_stat
20 tasks
jprestop
approved these changes
Apr 2, 2021
j-opatz
approved these changes
Apr 4, 2021
There was a problem hiding this comment.
The documentation changes sufficiently addressed the updates, and by listing it only for "-out_stat" it's implied that the other options do not repress the standard line output. This seems to meet the user's needs with minimal impact to the code, which should always be a goal when possible.
10 tasks
JohnHalleyGotway
added a commit
that referenced
this pull request
Apr 5, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Apr 12, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Apr 17, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> * feature_1552_gcc_10 (#1752) * Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * Feature 1735 stat_analysis (#1754) * Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds. * Per #1753, this one change to write_tmp_data.py solves this problem. When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose. * Per #1753, make the read_tmp_dataplane.py script less verbose. * Per #1753, there are 3 calls to the user's python version throughout MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise. * Per #1753, wrap the call to get_fill_value() in a try block in case the input in a regular numpy array instead of a masked array. Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Apr 27, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> * feature_1552_gcc_10 (#1752) * Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * Feature 1735 stat_analysis (#1754) * Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds. * Per #1753, this one change to write_tmp_data.py solves this problem. When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose. * Per #1753, make the read_tmp_dataplane.py script less verbose. * Per #1753, there are 3 calls to the user's python version throughout MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise. * Per #1753, wrap the call to get_fill_value() in a try block in case the input in a regular numpy array instead of a masked array. * Per #1620, correct bug in read_ascii_mpr.py script. The MPR line type has 37 columns in it, not 36! (#1760) * Feature 1700 python (#1765) * Per #1700, no real change, removing extra newline. * Per #1700, move global_python.h from vx_data2d_python over to the vx_python3_utils library where it belongs better. * Per #1700, no code changes. Just removing commented out code. * Per #1700, lots of little changes to make the python scripts consistent, updating the write*.py functions to add the user script directory to the system path, and remove extraneous log messages. * Per #1700, rename generic_python.py to set_python_env.py. Still actually need to change the source code to handle this change! * Per #1700 remove the pickle import. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, update read_tmp_ascii.py to work for both ascii2nc and stat_analysis. Just create an object named ascii_data and have both instances read it. * Per #1700, getting closer. Work in progress. Just need to get user-python embedding working for stat-analysis. * Per #1700, removing extraneous cout. * Per #1700, fix logic in PyLineDataFile::do_tmp_ascii() to get stat_analysis python embedding working again. * Per #1700, just comments. * Per #1700, replace references to pickle with user_python * Per #1700, update documentation to replace pickle with temp files. * Feature 1766 v10.0.0_beta5 (#1767) * Per #1766, udpate the release date and add release notes for v10.0.0-beta5. * Per #1766 and #1728, update the copyright notice year to 2021. Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
May 24, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> * feature_1552_gcc_10 (#1752) * Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * Feature 1735 stat_analysis (#1754) * Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds. * Per #1753, this one change to write_tmp_data.py solves this problem. When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose. * Per #1753, make the read_tmp_dataplane.py script less verbose. * Per #1753, there are 3 calls to the user's python version throughout MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise. * Per #1753, wrap the call to get_fill_value() in a try block in case the input in a regular numpy array instead of a masked array. * Per #1620, correct bug in read_ascii_mpr.py script. The MPR line type has 37 columns in it, not 36! (#1760) * Feature 1700 python (#1765) * Per #1700, no real change, removing extra newline. * Per #1700, move global_python.h from vx_data2d_python over to the vx_python3_utils library where it belongs better. * Per #1700, no code changes. Just removing commented out code. * Per #1700, lots of little changes to make the python scripts consistent, updating the write*.py functions to add the user script directory to the system path, and remove extraneous log messages. * Per #1700, rename generic_python.py to set_python_env.py. Still actually need to change the source code to handle this change! * Per #1700 remove the pickle import. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, update read_tmp_ascii.py to work for both ascii2nc and stat_analysis. Just create an object named ascii_data and have both instances read it. * Per #1700, getting closer. Work in progress. Just need to get user-python embedding working for stat-analysis. * Per #1700, removing extraneous cout. * Per #1700, fix logic in PyLineDataFile::do_tmp_ascii() to get stat_analysis python embedding working again. * Per #1700, just comments. * Per #1700, replace references to pickle with user_python * Per #1700, update documentation to replace pickle with temp files. * Feature 1766 v10.0.0_beta5 (#1767) * Per #1766, udpate the release date and add release notes for v10.0.0-beta5. * Per #1766 and #1728, update the copyright notice year to 2021. * Bugfix 1768 edeck (#1769) * Per #1768, update logic in ATCFProbLine::read_line(). If read_line() from the base class returns bad status, have this one return bad status as well. But do NOT for unsupported line types. Just print a Debug(4) log message instead. * Per #1768, update the probability line types to match those listed in https://www.nrlmry.navy.mil/atcf_web/docs/database/new/edeck.txt. That documentation was last updated in 11/2020, so presumably these reflect NHC's latest changes. * Per #1768, renaming enumerated value from ATCFLineType_ProbRIRW to ATCFLineType_ProbRI since there are now separated ATCF line type for rapid intensitifcation (RI) and weakening (RW). Will work more with this data in future issues to verify more of these probability types. * Feature 1771 release_notes (#1772) * Per #1771, draft version of combined met-10.0.0 release notes. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, also update the flowchart for met-10.0.0. * Per #1771, update flowchart to indicate that tc_gen now has netcdf output. * Per #1771, rotate the authorship list for met-10.0.0, shifting Barb from first author down to the end. * Committing hotfix to the develop branch to fix a bad merge that caused the MET compilation to fail. * Update compile_MET_all.sh Added "-L${LIB_LIBPNG}" to rpath to fix problem on WCOSS" * Update pull_request_template.md Added entry for completion date of pull request review. * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Per #1777, fixing memory management in DbfHeader::set_subrecords(). It is dynamically allocating a buffer based on the record_legnth (e.g. 5) but then reading 32 characters into it! Deleting the dynamically allocated buf variable causes it to abort. Since we always read 32 bytes here, switch to a static buffer of that size rather than dynamically allocating. (#1779) Co-authored-by: [email protected] <[email protected]> * Feature 1731 authorship (#1776) * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Updated input sources to include newly acceptable data formats * Per #1731, add more details about the grid-diag bin definition. * Per #1731, clarify wording. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Changes to align Xarray language with that used in Xarray documentation, to clarify only DataArray Xarray structures are supported and not Xarray Dataset structures, and an example of how to quickly create a DataArray for MET from a Dataset. * Corrects plural of DataArray. * Aligns references to NumPy arrays with NumPy docs to refer to them as ndarrays. * Fixes formatting of note for Xarray. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Fixes hyperlink reference. * Fixes line spacing in note directive. * Changes to reference again. * #1782 Set the time offset to 0 if the time dimension does not exist at the data variable * Updates to link and note directive. * Corrects plural of DataArray once more. * Adds more clarity to NumPy heading. * Adds bold for emphasis. * Removes bold in note directive. * Feature 1778 debug (#1785) * Per #1778, please see #1778 (comment) for details. Basically, when doing development, compile with the -g debug option. Otherwise, remove it by default. * Per #1778, update stale URL's in the README and configure.ac file. Also, change the default MET version from 8.1 to development. * Made suggested changes by Tara Jensen * Update python embedding docs to list required packages for the base python version. * Feature 1786 v10.0.0 (#1787) * Per #1786, updates for the v10.0.0 release. Note that no changes were needed in conf.py. It had already been updated. * Added update MET to compile using GNU version 10 compilers and PGI version 20 compilers * Made updates to improve compilation * Per #1768, added a couple more 10.0.0 release notes. * Adding pgi config file from cheyenne Co-authored-by: Julie.Prestopnik <[email protected]> * Per #1789, remove duplicate plot_point_obs configuration section. (#1791) * Update the version of Fortify on kiowa from 19.2.0 to 20.2.1. * #1795 Release memory at time_values * Bugfix 1798 develop py_grid_string (#1800) * Per #1798, fix up the read_tmpe_dataplane.py script to handle a grid string or dictionary. * Per #1798, add a test to unit_python.xml to exercise this bugfix. * #1794 Corrected the offset for Filter * Github Issue #1801: Comment out code that checks for BEST track to support extra-tropical cyclone tracks not verified against BEST tracks. * #1795 Cleanup * #1795 Create DataCube for 2D or 3D only, not both to avoid memory leak * Bugfix 1395 develop comp script (#1804) * Updated compile script and added assocaited config files * Added jet config file * Updated orion file * Added new stampede config file and modulefiles for various machines * Gitub Issue #1801 Remove code that checks for -bmodel filter to support plotting of extra-tropical cyclone tracks that aren't verified against BEST tracks. Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: bikegeek <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Jun 1, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> * feature_1552_gcc_10 (#1752) * Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * Feature 1735 stat_analysis (#1754) * Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds. * Per #1753, this one change to write_tmp_data.py solves this problem. When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose. * Per #1753, make the read_tmp_dataplane.py script less verbose. * Per #1753, there are 3 calls to the user's python version throughout MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise. * Per #1753, wrap the call to get_fill_value() in a try block in case the input in a regular numpy array instead of a masked array. * Per #1620, correct bug in read_ascii_mpr.py script. The MPR line type has 37 columns in it, not 36! (#1760) * Feature 1700 python (#1765) * Per #1700, no real change, removing extra newline. * Per #1700, move global_python.h from vx_data2d_python over to the vx_python3_utils library where it belongs better. * Per #1700, no code changes. Just removing commented out code. * Per #1700, lots of little changes to make the python scripts consistent, updating the write*.py functions to add the user script directory to the system path, and remove extraneous log messages. * Per #1700, rename generic_python.py to set_python_env.py. Still actually need to change the source code to handle this change! * Per #1700 remove the pickle import. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, update read_tmp_ascii.py to work for both ascii2nc and stat_analysis. Just create an object named ascii_data and have both instances read it. * Per #1700, getting closer. Work in progress. Just need to get user-python embedding working for stat-analysis. * Per #1700, removing extraneous cout. * Per #1700, fix logic in PyLineDataFile::do_tmp_ascii() to get stat_analysis python embedding working again. * Per #1700, just comments. * Per #1700, replace references to pickle with user_python * Per #1700, update documentation to replace pickle with temp files. * Feature 1766 v10.0.0_beta5 (#1767) * Per #1766, udpate the release date and add release notes for v10.0.0-beta5. * Per #1766 and #1728, update the copyright notice year to 2021. * Bugfix 1768 edeck (#1769) * Per #1768, update logic in ATCFProbLine::read_line(). If read_line() from the base class returns bad status, have this one return bad status as well. But do NOT for unsupported line types. Just print a Debug(4) log message instead. * Per #1768, update the probability line types to match those listed in https://www.nrlmry.navy.mil/atcf_web/docs/database/new/edeck.txt. That documentation was last updated in 11/2020, so presumably these reflect NHC's latest changes. * Per #1768, renaming enumerated value from ATCFLineType_ProbRIRW to ATCFLineType_ProbRI since there are now separated ATCF line type for rapid intensitifcation (RI) and weakening (RW). Will work more with this data in future issues to verify more of these probability types. * Feature 1771 release_notes (#1772) * Per #1771, draft version of combined met-10.0.0 release notes. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, also update the flowchart for met-10.0.0. * Per #1771, update flowchart to indicate that tc_gen now has netcdf output. * Per #1771, rotate the authorship list for met-10.0.0, shifting Barb from first author down to the end. * Committing hotfix to the develop branch to fix a bad merge that caused the MET compilation to fail. * Update compile_MET_all.sh Added "-L${LIB_LIBPNG}" to rpath to fix problem on WCOSS" * Update pull_request_template.md Added entry for completion date of pull request review. * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Per #1777, fixing memory management in DbfHeader::set_subrecords(). It is dynamically allocating a buffer based on the record_legnth (e.g. 5) but then reading 32 characters into it! Deleting the dynamically allocated buf variable causes it to abort. Since we always read 32 bytes here, switch to a static buffer of that size rather than dynamically allocating. (#1779) Co-authored-by: [email protected] <[email protected]> * Feature 1731 authorship (#1776) * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Updated input sources to include newly acceptable data formats * Per #1731, add more details about the grid-diag bin definition. * Per #1731, clarify wording. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Changes to align Xarray language with that used in Xarray documentation, to clarify only DataArray Xarray structures are supported and not Xarray Dataset structures, and an example of how to quickly create a DataArray for MET from a Dataset. * Corrects plural of DataArray. * Aligns references to NumPy arrays with NumPy docs to refer to them as ndarrays. * Fixes formatting of note for Xarray. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Fixes hyperlink reference. * Fixes line spacing in note directive. * Changes to reference again. * #1782 Set the time offset to 0 if the time dimension does not exist at the data variable * Updates to link and note directive. * Corrects plural of DataArray once more. * Adds more clarity to NumPy heading. * Adds bold for emphasis. * Removes bold in note directive. * Feature 1778 debug (#1785) * Per #1778, please see #1778 (comment) for details. Basically, when doing development, compile with the -g debug option. Otherwise, remove it by default. * Per #1778, update stale URL's in the README and configure.ac file. Also, change the default MET version from 8.1 to development. * Made suggested changes by Tara Jensen * Update python embedding docs to list required packages for the base python version. * Feature 1786 v10.0.0 (#1787) * Per #1786, updates for the v10.0.0 release. Note that no changes were needed in conf.py. It had already been updated. * Added update MET to compile using GNU version 10 compilers and PGI version 20 compilers * Made updates to improve compilation * Per #1768, added a couple more 10.0.0 release notes. * Adding pgi config file from cheyenne Co-authored-by: Julie.Prestopnik <[email protected]> * Per #1789, remove duplicate plot_point_obs configuration section. (#1791) * Update the version of Fortify on kiowa from 19.2.0 to 20.2.1. * #1795 Release memory at time_values * Bugfix 1798 develop py_grid_string (#1800) * Per #1798, fix up the read_tmpe_dataplane.py script to handle a grid string or dictionary. * Per #1798, add a test to unit_python.xml to exercise this bugfix. * #1794 Corrected the offset for Filter * Github Issue #1801: Comment out code that checks for BEST track to support extra-tropical cyclone tracks not verified against BEST tracks. * #1795 Cleanup * #1795 Create DataCube for 2D or 3D only, not both to avoid memory leak * Bugfix 1395 develop comp script (#1804) * Updated compile script and added assocaited config files * Added jet config file * Updated orion file * Added new stampede config file and modulefiles for various machines * Gitub Issue #1801 Remove code that checks for -bmodel filter to support plotting of extra-tropical cyclone tracks that aren't verified against BEST tracks. * Migrate issue and PR template changes from PR #1803 into the develop branch so that they'll be available for future releases. * Per met-help question (https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99964) clarify the description of the obs_thresh option. * Update README.md Adding GitHub Discussions information * changed non-unicode apostrophe and fixed typo in URL * Feature 1581 api point obs (#1812) * #1581 Initial release * #1581 Added met_nc_point_obs.cc met_nc_point_obs.h * Removed nc_ in function names and moved them to the struct members * #1581 Added HDR_TYPE_ARR_LEN * #1581 Changed API calls (API names) * #1581 Cleanup * #1581 Removed duplicated definitions:hdr_arr_len, hdr_typ_arr_len, and obs_arr_len * #1581 Removed duplicated definitions:hdr_arr_len and obs_arr_len * #1581 Removed duplicated definitions: str_len, hdr_arr_len, and obs_arr_len * Added vx_nc_obs library * #1581 Using common APIs * #1581 Corrected API calls because of renaming for common APIs * #1581 Moved function from nc_obs_util to nc_point_obs2 * #1581renamed met_nc_point_obs to nc_point_obs * #1581 API ica changed from obs_vars to nc_point_obs * #1581 Initial release * #1581 Renamed from met_nc_point_obs to nc_point_obs * 1581 Renamed met_nc_point_obs to nc_point_obs * Per #1581, update the Makefile.am for lidar2nc to fix a linker error. Unfortunatley, the vx_config library now depends on the vx_gsl_prob library. threshold.cc in vx_config includes a call to normal_cdf_inv(double, double, double) which is defined in vx_gsl_prob. This adds to the complexity of dependencies for MET's libraries. Just linking to -lvx_gsl_prob one more time does fix the linker problem but doesn't solve the messy dependencies. * #1581 Added method for NcDataBuffer * Cleanup * #1581 Cleanup * #1581 Cleanup * #1591 Cleanup * #1591 Corrected API * #1581 Avoid reading PB header twice * #1581 Warning if PB header is not defined but read_pb_hdr_data is called * #1581 Cleanup libraries * 1581 cleanup * 1581 cleanup * 1581 cleanup * #1581 Cleanup for Fortify (removed unused variables) * #1581 Cleanup * #1581 Cleanup * #1581 Use MetNcPointObsIn instead of MetNcPointObs * #1581 Use MetNcPointObsOut instead of MetNcPointObs2Write * #1581 Separated nc_point_obs2.cc to nc_point_obs_in.cc and nc_point_obs_out.cc * #1581 Renamed nc_point_obs2.cc to nc_point_obs_in.cc And added add nc_point_obs_in.h nc_point_obs_out.h nc_point_obs_out.cc * #1581 Removed APIs related with writing point obs * #1581 Changed copyright years * #1581 Cleanup * #1581 Updated copyright year * #1581 Cleanup * #1581 Reanmed read_obs_data_strings to read_obs_data_table_lookups * #1581 Reanmed read_obs_data_strings to read_obs_data_table_lookups * #1581 Added more APIs Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: George McCabe <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Jun 13, 2021
* Start on write netcdf pickle alternative. * Write dataplane array. * Start on read of netcdf as pickle alternative. * Create attribute variables. * Use global attributes for met_info attrs. * Add grid structure. * Read metadata back into met_info.attrs. * Convert grid.nx and grid.ny to int. * Rename _name key to name. * Removed pickle write. * Fixed write_pickle_dataplane to work for both numpy and xarray. * Use items() to iterate of key, value attrs. * Write temporary text file. * Renamed scripts. * Changed script names in Makefile.am. * Replaced pickle with tmp_nc. * Fixed wrapper script names. * Test for attrs in met_in.met_data. * Initial version of read_tmp_point module. * Added read_tmp_point.py to install list. * Start on Python3_Script::read_tmp_point. * Write MPR tmp ascii file. * Renamed to read_tmp_ascii to use for point point and MPR. * Renamed to read_tmp_ascii to use for point point and MPR. * Define Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::import_read_tmp_ascii_py. * Append MET_BASE/wrappers to sys.path. * Finished implementation of Python3_Script::import_read_tmp_ascii_py. * Call Python3_Script::read_tmp_ascii in python_handler. * Revised python3_script::read_tmp_ascii with call to run, PyRun_String. * Return PyObject* from Python3_Script::run. * Restored call to run_python_string for now. * Per #1429, enhance error message from DataLine::get_item(). (#1682) * Feature 1429 tc_log second try (#1686) * Per #1429, enhance error message from DataLine::get_item(). * Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message. * Feature 1588 ps_log (#1687) * Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good. * Per #1588, switch very detailed interpolation details from debug level 4 to 5. * Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level. * Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs. * #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced * #1454 Moved NC attribute name to nc_utils.h * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance * #1454 Corrected data.delta_lon * #1454 Change bact to use diff instead of absolute value of diff * 454 Deleted instea dof commenting out * 454 Deleted instea dof commenting out * Feature 1684 bss and 1685 single reference model (#1689) * Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions. * Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class. * Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file. * Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions. * Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic. * Per #1684, during development, I ran across and then updated this log message. * Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2). * Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values. * Per #1684, just fixing the format of this log message. * Per #1684, add a STATLine::get_offset() member function. * Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number. * Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones. * Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution. * Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model. * Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects. * Per #1684, remove debug print statements. * Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis. * Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference: For a climo distribution defined by mean and stdev: DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble. For a single deterministic reference: DEBUG 3: Computing ensemble statistics relative to the climatological mean. * Per #1691, add met-10.0.0-beta4 release notes. (#1692) * Updated Python documentation * Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions. * Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place. * Per #1694, just switching to consistent variable name. * Just consistent spacing. * Added python3_script::import_read_tmp_ascii. * Restored read_tmp_ascii call. * Added lookup into ascii module. * Adding files for ReadTheDocs * Adding .yaml file for ReadTheDocs * Updated path to requirements.txt file * Updated path to conf.py file * Removing ReadTheDocs files and working in separate branch * Return PyObject* from read_tmp_ascii. * Put point_data in global namespace. * Remove temporary ascii file. * Added tmp_ascii_path. * Removed read_obs_from_pickle. * Trying different options for formats (#1702) * Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708) * Feature 1471 python_grid (#1704) * Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics. * Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution. * Per #1471, add a sanity check to make sure the grid and data dimensions actually match. * Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file. * Per #1471, document python grid changes in appendix F. * Per #1471, just spacing. * Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic. Co-authored-by: John Halley Gotway <[email protected]> * Committing a fix for unit_python.xml directly to the develop branch. We referenced in a place where it's not defined. * Add *.dSYM to the .gitignore files in the src and internal_tests directories. * Replaced tmp netcdf _name attribute with name_str. * Append user script path to system path. * Revert "Feature 1319 no pickle" (#1717) * Fixed typos, added content, and modified release date format * #1715 Initial release * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added pb2nc_compute_pbl_cape * #1715 Added pb2nc_compute_pbl_cape * #1715 Reduced obs_bufr_var. Removed pb_report_type * #1715 Added a blank line for Error/Warning * Per #1725, return good status from TrackInfoArray::add() when using an ATCF line to create a new track. (#1726) * Per #1705, update the threshold node heirarchy by adding a climo_prob() function to determine the climatological probability of a CDP-type threshold. Also update derive_climo_prob() in pair_base.cc to call the new climo_prob() function. (#1724) * Bugfix 1716 develop perc_thresh (#1722) * Per #1716, committing changes from Randy Bullock to support floating point percentile thresholds. * Per #1716, no code changes, just consistent formatting. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Update pull_request_template.md * Feature 1733 exc (#1734) * Per #1733, add column_exc_name, column_exc_val, init_exc_name, and init_exc_val options to the TCStat config files. * Per #1733, enhance tc_stat to support the column_exc and init_exc config file and job command filtering options. * Per #1733, update stat_analysis to support the -column_exc job filtering option. Still need to update docuementation and add unit tests. * Per #1773, update the user's guide with the new config and job command options. * Per #1733, add call to stat_analysis to exercise -column_str and -column_exc options. * Per #1733, I ran into a namespace conflict in tc_stat where -init_exc was used for to filter by time AND my string value. So I switched to using -init_str_exc instead. And made the corresponding change to -column_str_exc in stat_analysis and tc_stat. Also changed internal variable names to use IncMap and ExcMap to keep the logic clear. * Per #1733, tc_stat config file updates to switch from column_exc and init_exc to column_str_exc and init_str_exc. * Per #1733, add tc_stat and stat_analysis jobs to exercise the string filtering options. * Bugfix 1737 develop little_r (#1739) * Per #1737, migrate the same fix from main_v9.1 over to the develop branch. * Per #1737, add another unit test for running ascii2nc with corrupt littl_r records. * Feature GitHub actions (#1742) * Adding files to build documenation via GitHub Actions * Removing html_theme_options * Removed warnings.log from help section * Feature 1575 large_diffs (#1741) * Per #1575, add mpr_column and mpr_thresh entries to all of the Grid-Stat and Point-Stat config files. * Per #1575, define config strings to be parsed from the config files. * Per #1575, store col_name_ptr and col_thresh_ptr in PairBase. They are being used for PairDataPoint to do MPR filtering in Grid-Stat and Point-Stat. But they could be eventually be extended to filter ORANK columns for Ensemble-Stat. * Per #1575, add MPR filtering logic to pair_data_point.cc. Include filtering logic in PairDataPoint instead of VxPairDataPoint since Grid-Stat uses PairDataPoint. * Per #1575, update point_stat to parse the mpr_column and mpr_thresh config file options. Include the MPR rejection reason code counts in the log output. * Per #1575, updated Grid-Stat to parse mpr_column and mpr_thresh options. * Per #1575, update Point-Stat to store mpr_sa and mpr_ta locally and then call set_mpr_filt() after the VxPairDataPoint object has been sized and allocated. * Per #1575, renamed PairDataEnsemble::subset_pairs() to subset_pairs_obs_thresh() to be a little more explicit about things. I'll do the same for PairDataPoint using names subset_pairs_cnt_thresh() and subset_pairs_mpr_thresh(). * Per #1575, some cleanup, moving check_fo_thresh() utility function from vx_config to vx_statistics library. * Per #1575, when implementing this for Grid-Stat, I realized that there isn't much benefit in storing col_name_ptr and col_name_thresh in PairBase. These changes remove that. * Per #1575, updating pair_data_point.h/.cc to handle the subsetting of data based on the MPR thresh. * Per #1575, rename subset_pairs() to subset_pairs_cnt_thresh() to be a bit more explicit with the naming conventions. * Per #1575, no real changes here. Just reorganizing the location of the mpr_sa and mpr_ta members. * Per #1575, make the subset_pairs() utility function a member function of the PairDataPoint class named subset_pairs_cnt_thresh() and update the application code to call it. * Per #1575, need to actually set the mpr_thresh! * Per #1575, update subset_pairs_mpr_thresh() to make sure the StringArray and ThreshArray lengths are the same. * Per #1575, replace PairDataPoint::subset_pairs_mpr_thresh() with a utility function named apply_mpr_thresh_mask(). This is for Grid-Stat to apply the mpr_thresh settings after the DataPlane pairs have been created but prior to applying any smoothing operations. * Per #1575, add documentation about mpr_column and mpr_thresh. * Per #1575, mpr_columns can also include CLIMO_CDF. * Per #1575, add tests for Grid-Stat and Point-Stat to exercise the mpr_column and mpr_thresh config file options. * Feature 1319 no pickle (#1720) * Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1736 out_stat (#1744) * Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <[email protected]> * Per #1319, this is a hotfix to the develop branch. While running unit_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test. * Just lining up a log message in the output of gen_vx_mask. * Per #1319, setting PATH as an envvar might cause problems. All variables set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA. * Trying to get the PATH setting correct for unit_python.xml. * Changed weblink for METplus documentation * per #1319 added netCDF4 python package to MET docker image so it is available for python embedding cases that use MET_PYTHON_EXE * Feature 1747 pylonglong (#1748) * Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working. * Hotfix for the develop branch in tc_pairs.cc. The METplus unit tests kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead. * Turned specific section numbers into linked sections because section numbers can change * Removed hard-coded references to section numbers * Feature 1714 tc_gen (#1750) * Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <[email protected]> * feature_1552_gcc_10 (#1752) * Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * #1755 Support time string for slicing at MetNcCFDataFile::collect_time_offsets * Feature 1735 stat_analysis (#1754) * Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds. * Per #1753, this one change to write_tmp_data.py solves this problem. When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose. * Per #1753, make the read_tmp_dataplane.py script less verbose. * Per #1753, there are 3 calls to the user's python version throughout MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise. * Per #1753, wrap the call to get_fill_value() in a try block in case the input in a regular numpy array instead of a masked array. * Per #1620, correct bug in read_ascii_mpr.py script. The MPR line type has 37 columns in it, not 36! (#1760) * Feature 1700 python (#1765) * Per #1700, no real change, removing extra newline. * Per #1700, move global_python.h from vx_data2d_python over to the vx_python3_utils library where it belongs better. * Per #1700, no code changes. Just removing commented out code. * Per #1700, lots of little changes to make the python scripts consistent, updating the write*.py functions to add the user script directory to the system path, and remove extraneous log messages. * Per #1700, rename generic_python.py to set_python_env.py. Still actually need to change the source code to handle this change! * Per #1700 remove the pickle import. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, work in progress. Replaced pickle file with tmp file. * Per #1700, update read_tmp_ascii.py to work for both ascii2nc and stat_analysis. Just create an object named ascii_data and have both instances read it. * Per #1700, getting closer. Work in progress. Just need to get user-python embedding working for stat-analysis. * Per #1700, removing extraneous cout. * Per #1700, fix logic in PyLineDataFile::do_tmp_ascii() to get stat_analysis python embedding working again. * Per #1700, just comments. * Per #1700, replace references to pickle with user_python * Per #1700, update documentation to replace pickle with temp files. * Feature 1766 v10.0.0_beta5 (#1767) * Per #1766, udpate the release date and add release notes for v10.0.0-beta5. * Per #1766 and #1728, update the copyright notice year to 2021. * Bugfix 1768 edeck (#1769) * Per #1768, update logic in ATCFProbLine::read_line(). If read_line() from the base class returns bad status, have this one return bad status as well. But do NOT for unsupported line types. Just print a Debug(4) log message instead. * Per #1768, update the probability line types to match those listed in https://www.nrlmry.navy.mil/atcf_web/docs/database/new/edeck.txt. That documentation was last updated in 11/2020, so presumably these reflect NHC's latest changes. * Per #1768, renaming enumerated value from ATCFLineType_ProbRIRW to ATCFLineType_ProbRI since there are now separated ATCF line type for rapid intensitifcation (RI) and weakening (RW). Will work more with this data in future issues to verify more of these probability types. * Feature 1771 release_notes (#1772) * Per #1771, draft version of combined met-10.0.0 release notes. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, add bolding to indicate emphasis. * Per #1771, also update the flowchart for met-10.0.0. * Per #1771, update flowchart to indicate that tc_gen now has netcdf output. * Per #1771, rotate the authorship list for met-10.0.0, shifting Barb from first author down to the end. * Committing hotfix to the develop branch to fix a bad merge that caused the MET compilation to fail. * Update compile_MET_all.sh Added "-L${LIB_LIBPNG}" to rpath to fix problem on WCOSS" * Update pull_request_template.md Added entry for completion date of pull request review. * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Per #1777, fixing memory management in DbfHeader::set_subrecords(). It is dynamically allocating a buffer based on the record_legnth (e.g. 5) but then reading 32 characters into it! Deleting the dynamically allocated buf variable causes it to abort. Since we always read 32 bytes here, switch to a static buffer of that size rather than dynamically allocating. (#1779) Co-authored-by: [email protected] <[email protected]> * Feature 1731 authorship (#1776) * Per #1731, add ioda2nc documentation. * Changes to make the authorship list consistent with METplus. * Rename CIR to CIRA. * Per #1731, fix alignment issued caused by tabs vs spaces. * Updated input sources to include newly acceptable data formats * Per #1731, add more details about the grid-diag bin definition. * Per #1731, clarify wording. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Changes to align Xarray language with that used in Xarray documentation, to clarify only DataArray Xarray structures are supported and not Xarray Dataset structures, and an example of how to quickly create a DataArray for MET from a Dataset. * Corrects plural of DataArray. * Aligns references to NumPy arrays with NumPy docs to refer to them as ndarrays. * Fixes formatting of note for Xarray. * Update met/docs/Users_Guide/tc-pairs.rst Co-authored-by: johnhg <[email protected]> * Fixes hyperlink reference. * Fixes line spacing in note directive. * Changes to reference again. * #1782 Set the time offset to 0 if the time dimension does not exist at the data variable * Updates to link and note directive. * Corrects plural of DataArray once more. * Adds more clarity to NumPy heading. * Adds bold for emphasis. * Removes bold in note directive. * Feature 1778 debug (#1785) * Per #1778, please see #1778 (comment) for details. Basically, when doing development, compile with the -g debug option. Otherwise, remove it by default. * Per #1778, update stale URL's in the README and configure.ac file. Also, change the default MET version from 8.1 to development. * Made suggested changes by Tara Jensen * Update python embedding docs to list required packages for the base python version. * Feature 1786 v10.0.0 (#1787) * Per #1786, updates for the v10.0.0 release. Note that no changes were needed in conf.py. It had already been updated. * Added update MET to compile using GNU version 10 compilers and PGI version 20 compilers * Made updates to improve compilation * Per #1768, added a couple more 10.0.0 release notes. * Adding pgi config file from cheyenne Co-authored-by: Julie.Prestopnik <[email protected]> * Per #1789, remove duplicate plot_point_obs configuration section. (#1791) * Update the version of Fortify on kiowa from 19.2.0 to 20.2.1. * #1795 Release memory at time_values * Bugfix 1798 develop py_grid_string (#1800) * Per #1798, fix up the read_tmpe_dataplane.py script to handle a grid string or dictionary. * Per #1798, add a test to unit_python.xml to exercise this bugfix. * #1794 Corrected the offset for Filter * Github Issue #1801: Comment out code that checks for BEST track to support extra-tropical cyclone tracks not verified against BEST tracks. * #1795 Cleanup * #1795 Create DataCube for 2D or 3D only, not both to avoid memory leak * Bugfix 1395 develop comp script (#1804) * Updated compile script and added assocaited config files * Added jet config file * Updated orion file * Added new stampede config file and modulefiles for various machines * Gitub Issue #1801 Remove code that checks for -bmodel filter to support plotting of extra-tropical cyclone tracks that aren't verified against BEST tracks. * Migrate issue and PR template changes from PR #1803 into the develop branch so that they'll be available for future releases. * Per met-help question (https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99964) clarify the description of the obs_thresh option. * Update README.md Adding GitHub Discussions information * changed non-unicode apostrophe and fixed typo in URL * Feature 1581 api point obs (#1812) * #1581 Initial release * #1581 Added met_nc_point_obs.cc met_nc_point_obs.h * Removed nc_ in function names and moved them to the struct members * #1581 Added HDR_TYPE_ARR_LEN * #1581 Changed API calls (API names) * #1581 Cleanup * #1581 Removed duplicated definitions:hdr_arr_len, hdr_typ_arr_len, and obs_arr_len * #1581 Removed duplicated definitions:hdr_arr_len and obs_arr_len * #1581 Removed duplicated definitions: str_len, hdr_arr_len, and obs_arr_len * Added vx_nc_obs library * #1581 Using common APIs * #1581 Corrected API calls because of renaming for common APIs * #1581 Moved function from nc_obs_util to nc_point_obs2 * #1581renamed met_nc_point_obs to nc_point_obs * #1581 API ica changed from obs_vars to nc_point_obs * #1581 Initial release * #1581 Renamed from met_nc_point_obs to nc_point_obs * 1581 Renamed met_nc_point_obs to nc_point_obs * Per #1581, update the Makefile.am for lidar2nc to fix a linker error. Unfortunatley, the vx_config library now depends on the vx_gsl_prob library. threshold.cc in vx_config includes a call to normal_cdf_inv(double, double, double) which is defined in vx_gsl_prob. This adds to the complexity of dependencies for MET's libraries. Just linking to -lvx_gsl_prob one more time does fix the linker problem but doesn't solve the messy dependencies. * #1581 Added method for NcDataBuffer * Cleanup * #1581 Cleanup * #1581 Cleanup * #1591 Cleanup * #1591 Corrected API * #1581 Avoid reading PB header twice * #1581 Warning if PB header is not defined but read_pb_hdr_data is called * #1581 Cleanup libraries * 1581 cleanup * 1581 cleanup * 1581 cleanup * #1581 Cleanup for Fortify (removed unused variables) * #1581 Cleanup * #1581 Cleanup * #1581 Use MetNcPointObsIn instead of MetNcPointObs * #1581 Use MetNcPointObsOut instead of MetNcPointObs2Write * #1581 Separated nc_point_obs2.cc to nc_point_obs_in.cc and nc_point_obs_out.cc * #1581 Renamed nc_point_obs2.cc to nc_point_obs_in.cc And added add nc_point_obs_in.h nc_point_obs_out.h nc_point_obs_out.cc * #1581 Removed APIs related with writing point obs * #1581 Changed copyright years * #1581 Cleanup * #1581 Updated copyright year * #1581 Cleanup * #1581 Reanmed read_obs_data_strings to read_obs_data_table_lookups * #1581 Reanmed read_obs_data_strings to read_obs_data_table_lookups * #1581 Added more APIs Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1792 gen_vx_mask (#1816) * Per issue #1792, Change -type from optional to required. Set default_mask_type to MaskType_None. Added a check on mask_type to see if it's set and print error message accordingly. * Update test_gen_vx_mask.sh * For the first test, added -type poly, since the masking type is now required. SL * For all of the Poly unit tests added -type poly to the command line. The mask type is now required. SL * Modified document to indicate that -type string (masking type) is now required on the command line for gen_vx_mask. SL * Update met/docs/Users_Guide/masking.rst Co-authored-by: johnhg <[email protected]> * Update met/src/tools/other/gen_vx_mask/gen_vx_mask.cc Co-authored-by: johnhg <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: johnhg <[email protected]> * Fix 2 minor formatting errors in the release notes. * PR #1816 for issue #1792 unexpectedly caused the NB to fail on 20210605. We changed -type from optional to required, but missed adding the -type option in unit_met_test_scripts.xml and unit_ref_config.xml. This is a hotfix to resolve that. * Feature 1811 anchor links (#1822) * testing new anchoring link idea. * testing without the bold asterik * tinkering with the look * another attempt * another attempt #2 * another attempt #3 * another attempt #4 * making sure new anchor works as expected. * seeing if link will save with spaces instead of dashes * need underscores to link * is it fixed? * testing * testing 2 * testing 4 * testing 5 * testing 6 * testing 7 * testing 8 * going back to test original problem * able to link with spaces instead of underscores. Testing if a return is possible to keep under 79 character limit. * double checking everything is still working. * DO NOT break ref lines apart, it won't work. * trying a shorter name. * continuing to add anchors * updating lines 1946 thru 2214 with anchors * updating lines 2214 thru 3371 with anchors * updating lines 3371 to the end with anchors * testing anchor * testing anchor * testing anchor 3 * testing anchor 4 * testing anchor 45 percent * testing anchor final half * fixing typo * numbering fcst, obs_1 and 2 to create different links. * finding more anchors that need numbers to keep them separate. * fixing warnings * fixing warnings * fixing typo * Feature 1749 hss (#1825) * Per #1749, updating the MET version number from 10.0 to 10.1 prior to adding new columns of output to existing line types. * Per #1749, adding 10.1 columns to the Makefile.am * Per #1749, changes for the mechanics of adding the HSS_EC statistic to the MCTS line type. Still need to acutally compute it and make the expected correct value configurable. * Per #1749, add hss_ec_value as a configurable option for Point-Stat and Grid-Stat. Still need to actually compute it correctly, add it to other test config files, add support to series_analysis/stat_analysis, update the docs, and make writeup corresponding issues for other METplus components. * Per #1749, fix the column offsets for the HSS_EC columns. * Per #1749, add correct definition of HSS_EC. * Per #1749, pass hss_ec_value from the config file into the computation of the MCTS statistics. * Per #1749, add hss_ec_value entry to all the Grid-Stat config files. * Per #1749, update the documentation about the HSS_EC statistic. * Per #1749, add the -hss_ec_value job command option to Stat-Analysis. * Per #1749, no real code changes here. Just changing to consistent ordering with hss_ec_value preceeding rank_corr_flag. * Per #1749, update docs for stat_analysis supporting hss_ec_value. * Per #1749, add HSS_EC to Series-Analysis, but only with a constant hss_ec_value for now. * Per #1749, add EC_VALUE to the MCTC line type definition. * Per #1749, move ECvalue from the MCTSInfo class into the ContingencyTable class so that it's available to be included in the MCTC output line type. * Per #1749, update point_stat, grid_stat, and series_analysis to accomodate the move of ECvalue from the MCTSInfo class to the ContingencyTable class. * Per #1749, update library code to write EC_VALUE to the MCTC line type and update the User's Guide docs. * Per #1749, update stat_analysis code for the addition of EC_VALUE in the MCTC line type. * Per #1749, write EC_VALUE to the MCTC output line type. * Per #1749, store the ec_value that was actually used to compute the stats. * Per #1749, parsing EC_VALUE from the MCTC line type. * Per #1749, move the MCTC EC_VALUE column to the end of the line, as requested by METdatadb. * Per #932, need to write MCTS HSS_EC value to temp file during the bootstrapping process. * Added new reference for Ou 2016 * Layout correction * Added generalized HSS, removed word from HSS_EC * Per #1749, change the hss_ec_value config entry to match new conventions. Co-authored-by: j-opatz <[email protected]> * Feature 1826 v10.1.0_beta1 (#1828) * Per #1826, add update the version in the docs to 10.1.0-beta1 and add release notes for this development version. * Per #1826, change the beta1 release date to 6/11 so that I can do it today. * Revoming Randy and David from the email notification list for nightly run scripts. Co-authored-by: David Fillmore <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Julie.Prestopnik <[email protected]> Co-authored-by: David Fillmore <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: lisagoodrich <[email protected]>
jprestop
added a commit
that referenced
this pull request
Sep 2, 2022
JohnHalleyGotway
added a commit
that referenced
this pull request
Nov 3, 2022
* Initialze obs_data at constructor to avoid warninjg on building * Initialze obs_data at constructor to avoid a warning by compiler * #1824 ci-run-test Removed the duplicated code * Feature 1810 expand asciitable (#2086) * Feature 1583 skip_mean (#2090) * Feature 1275 MODE Object Count (#2091) * for #1275, fix issue with more than 1000 object for mode, ci-run-unit * Per #1275, just deleting stale, commented-out code. Co-authored-by: Randy Bullock <[email protected]> * Feature 1184 dryline (#2088) Co-authored-by: davidfillmore <[email protected]> Co-authored-by: rgbullock <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1184 replace_env (#2093) * Feature 2092 v10.1.0-rc1 (#2094) * Hotfix to the develop branch to fix the MODE Makefile for the MET-10.1.0-rc1 release. * Second hotfix for the MODE Makefile in the develop branch. * Feature 1844 docs (#2096) * Feature 2097 v10.1.0 (#2099) * Feature 2098 gen ens prod doc (#2100) * update job control rules for GHA testing workflow to use version specific input data if running tests on main_v branch * Migrating a small tweak added to the v10.1.0 release into the develop branch. * Adding SonarQube location to development.seneca file. * Bugfix #2102 develop initialize modified_hdr_typ (#2104) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: johnhg <[email protected]> * added logic to manually trigger a workflow via the GitHub… (develop) (#2109) Co-authored-by: John Halley Gotway <[email protected]> * Feature update installation info (#2114) * Updating configuration and modulefiles * Removed jet files * create DockerHub tag that ends with -lite to use when triggering the METplus testing workflow * Bugfix #2115 develop Rotated LatLon (#2117) * Feature 1950 sort station list (#2121) * Update the top-level README file to list ioda2nc, but also to trigger GHA to run for this new main_v10.1-ref branch. * Added bool Sorted variable, set to true or false in relevant functions. Updated sort() function. SL * In parse_sid_mask(), added the line: mask_sid.sort(). SL * Per issue #1950, put in some cout statements for testing. SL * Per issue #1950, working in has() function, checking for Sort and put in initial cout's for testing. SL * Per issue #1950: Cleaned up some cout (print) statements. Left one in there that is commented out for future testing. SL * Per issue #1950: modified has() function. Added lower_bound search for sorted arrays. Left some print (cout) statements in there for further testing. SL * Per issue #1950: In has(), I commented out the print (cout) statements for time testing. SL * added logic to manually trigger a workflow via the GitHub… (#2107) * Per issue #1950, put back in some print (cout) statements for testing. SL * Per issue #1950: in has() function, experimenting using the upper_bound() search instead of lower_bound() for sorted vectors. In progress. SL * Per issue #1950: added a new has function that uses binary_search. Re-working the other has() functions related to this. SL * Per issue #1950: re-worked original has(), basically reverted back to what it was...but it's now only used for unsorted vectors or for a case insensitive search. Cleaned up. SL * Per issue #1950: in section that checks Obs Station Id in SID masking list, changed cout to a mlog (debug) statement. SL * Per issue #1950: created new mask station id list (CONUS METARs) and modified the PointStatConfig_MASK_SID config file to use this list along with the existing lists. SL * Per issue #1950: reverted back to orginal sid mask site-lists. SL * Per issue #1950: checking in this site-list that was used for testing point_stat. SL * Per issue #1950, modified and added better debugging to figure out current issue with ascii2nc. SL * Per #1950, small tweak in ascii2nc to make it slightly more efficient. After adding the new obs varable name to the end of the list, we can compute var_index direclty without needing to call StringArray::has() a second time. * Per #1950: removed mlog and cout print statements from has() functions. Cleaned up. SL * Update met/src/basic/vx_log/string_array.cc Co-authored-by: johnhg <[email protected]> * Update met/src/basic/vx_log/string_array.cc Co-authored-by: johnhg <[email protected]> * Update met/src/basic/vx_log/string_array.cc Co-authored-by: johnhg <[email protected]> * Update met/src/basic/vx_log/string_array.cc Co-authored-by: johnhg <[email protected]> * Update met/src/basic/vx_log/string_array.cc Co-authored-by: johnhg <[email protected]> * Update met/src/basic/vx_log/string_array.h Co-authored-by: johnhg <[email protected]> * Delete SID_CONUS.txt * Delete SID_CONUS_times11.txt Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2118 develop grib1_rotll (#2130) Co-authored-by: George McCabe <[email protected]> * Bugfix 2123 nccf valid time (#2127) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: johnhg <[email protected]> * Feature #2132 version 11.0.0 (#2133) * Bugfix #2106 develop gcc (#2135) Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: johnhg <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: Randy Bullock <[email protected]> Co-authored-by: davidfillmore <[email protected]> Co-authored-by: rgbullock <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: mo-mglover <[email protected]> Co-authored-by: davidalbo <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> * Doc-only update to expliciltly list the maximum character lengths for variable-length strings passed to ascii2nc. * Added pcp_combine -subtract example (#2153) * Add testing and docs status badges. * Feature #2152 poly_xy (#2154) * #14 Using literal instead of numbers and make sure no overflow * #14 Using literal instead of numbers * #14 Added tmp_buf_size * #14 Define n_kw_infos first * #14 Using literal instead of numbers * Bugfix #2148 develop misses (#2150) * dtcenter/METplus-Internal#14 formatting code * METplus-Internal #16 Added python scanning * METplus-Internal #16 changed name * METplus-Internal #16 initial release for python scanning * METplus-Internalv#19 added maic.cc & main.h * METplus-Internalv#19 renamed main to met_main and added get_tool_name * METplus-Internalv#19 Initial release * changing 'Linked issue' to 'Development issue' * change Linked issue/Development issue ci-skip-all * change Linked issue/Development issue ci-skip-all * change Linked issue/Development issue ci-skip-all * Feature 1920 repo (#2161) * List line types that can be aggregated per dtcenter/METplus#1625 * Hotfix for the develop branch to correct a bad MET_TEST_BASE path that causes the nightly build to fail. * Feature 2072 tc rmw winds (#2165) * bug fix for NAN and for using missing data as if its not missing * added parameters to control converting winds to tangential/radial components * bug fixes * Refactored to put wind conversion into it's own class TCRMW_WindConverter * changed default for converting winds to TRUE, and put parameters for wind conversion into the unit test config files * Per #2072, updating tc_rmw Makefile.in to compile new files. * Per #2072, update to the output from the lex/yacc version on seneca. * Changed Error to Warning in mlog messages, for consistency with MET conventions * Removing commented out code. * Per #2072, combine warnings spread across multiple lines into a single call to the logger. Replace references to ugrid/vgrid with u/v wind. Fix a typo and clean up some whitespace. Co-authored-by: Dave Albo <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * change Linked issue/Development issue ci-skip-all * Bugfix #2173 develop shift_data_plane (#2175) * Bugfix #2170 develop agg_ecnt (#2177) * Feature 2022 anom corr to vcnt (#2171) * Per #2022: Added ANOM_CORR, ANOM_CORR_BCL, ANOM_CORR_BCU to vcnt_columns. SL * Per #2022: Added ANOM_CORR, ANOM_CORR_BCL, ANOM_CORR_BCU to write_vcnt_cols. SL * Per #2022: Added ANOM_CORR to VL1L2Info. SL * Per #2022: updated the VL1L2Info functions to initialize and compute ANOM_CORR. SL * Per #2022: in calc_ncep_stats modified calculation for ANOM_CORR to check for bad data in the denominator first. SL * Per #2022: for the VL1L2Info, changed all derived VCNT variables to be CIInfo objects (instead of doubles). SL * Per #2022: making changes to VL1L2Info related to making all VCTN variables CIInfo objects. In progress. SL * Per #2022: Added ANOM_CORR_UNCNTR. SL * Per issue 2022: Modified functions in the VL1L2Info class to set the NCEP stat variables correctly since they are now CIInfo class memembers. SL * Per issue 2022: Added ANOM_CORR_UNCNTR (_BCL, _BCU). SL * Per issue 2022: modified write_vcnt_cols(), added ANOM_CORR_UNCNTR (_BCL, _BCU). SL * Per issue #2022: updated write_vcnt_cols. VCNT variables are now CIInfo objects to have to .v for the value. SL * Per issue #2022: In calc_ncep_stats, put in place-holder for ANOM_CORR.v value SL * Per issue #2022: in calc_ncep_stats(), changed placeholder for ANOM_CORR. SL * Per issue #2022: moved alpha initialization to the init_from_scratch() function. SL * Per issue #2022: in calc_ncep_stats(), set ANOM_CORR to bad_data_double as a placeholder for now. SL * Per issue #2022: in vcnt_cols(), added ANOM_CORR_NCL/NCU. SL * Per issue #2022: in write_vcnt_cols(), added ANOM_CORR_NCL/NCU variables. SL * Per #2022, adding new VCNT columns to the header table files, User's Guide documentation, and lining some code up more consistently. * Per #2022, added FA_SPEED_BAR and OA_SPEEED_BAR to the VAL1L2 line type. Added code to compute the ANOM_CORR confidence interval. However, I still need to update the writing of the VCNT line type to set the alpha value and actually write the computed CI's. * Per #2022, update the writing of the VCNT output line to write the alpha value and anomaly correlation CI's. * Per #2022, update point_stat and grid_stat to allocate alpha values for VL1L2Info objects to store CI's. * Per #2022, call VL1L2Info::zero_out() instead of clear() so that we don't throw away allocated alpha values. * Per #2022, update the Point-Stat and Grid-Stat logic to include the number of alpha values when deciding how many VCNT lines to write. * Per #2022, update the VL1L2Info::operator+= logic to copy over the allocated alpha values. * Per #2022, enhance stat_analysis to support an aggregate_stat job type for VAL1L2->VCNT. This writes ANOM_CORR and ANOM_CORR_UNCNTR output columns. However, there are still things to address. The other stats should be written as NA instead of 0. Recommend changing the VL1L2Info::zero_out() function to intialize stat values to NA instead of 0. Need to update the docs to reflect this support. Also need to actually compute the parametric CI's for ANOM_CORR. Big picture, we should store the VCNT stats in a new VCNTInfo class... just like the SL1L2Info/CNTInfo setup. Storing the sums AND stats in the same VL1L2Info class is unnecessarily confusing. * Per #2022, update Stat-Analysis docs to list VAL1L2 -> VCNT as a supported conversion type. * Per #2022, when computing VL1L2Info CI's, n should be set to vacount rather than the number of input pairs. * Per #2022, add a new call to Stat-Analysis to exercise the VAL1L2 to VCNT conversion. * Per #2022, ci-run-unit fix an aggregation bug in VL1L2Info::operator+=(). * Per issue #2022, added _vcnt.txt output files to check to relevant tests based on if vcnt output file is being generated via the config file. SL Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * #1815 Renamed time_as_offset to Is_offset * #1815 Added vx_data2d_dim_by_value * #1815 Added convert_value_to_offset and renamed _time_dim_offset to _cur_time_index * #1815 Added convert_value_to_offset and renamed _time_dim_offset to _cur_time_index. Support value instead of offset for time and z level * #1815 Support @ for the value insteaed of offset for slicing * #1815 Support value insteaed of offset for slicing * #1815 Deleted unused local variable * #1815 Deleted unused local variable * #1815 Added level by value with @ * #1815 Added plot_data_plane_NCCF_latlon_by_value * #1815 Added level by value with @ * Feature #2180 develop kft (#2181) * dtcenter/METplus-Internal#16 Moved from scripts/sonarqube to internal/scripts/sonarqube * #1815 Updated level * Feature 2022 stat_analysis (#2185) Proceeding with the merge without an external review based on the details listed above. * Feature #2147 HSS (#2169) * #1815 call is_eq to comapre two double values * Per #2147, this is a HOTFIX to the develop branch after PR #2169 added new CTC and CTS columns. Prevoiusly the CTC columns = NBRCTC columns and CTS = NBRCTS. But that is no longer the case. This updates write_nbrctc_cols() and write_nbrcts_cols() to actually write them rather than just calling the functions to write CTC and CTS cols, respectively. * #2187 Added add_offset_att_name & scale_factor_att_name * #2187 Apply add_offse or scale_factor if defined * ##2187 Removed handling add_offset and scale factor. They are applied at the base API * #2187 Changed a log message * Per #2193, add release notes for met-11.0.0-beta1. (#2195) Co-authored-by: MET Tools Test Account <[email protected]> * Hotfix to develop since I forgot to update the version number in conf.py. * #1815 Added more attribute names * #1815 Added get_index_for_dim * #1815 Formatting * #1815 Support a differnt dimension name from thw first data variable * #1815 Use constants for attr names * Hotfix to the develop branch to fix paths in Dockefiles after restructuing the MET repo. * Another hotfix to develop to get the Docker build step working again. The failure from the configure script is caused by the version of the "alocal" executable. The contents of develop are based on seneca, which has: aclocal (GNU automake) 1.16.1 By comparison, my laptop does NOT require that bootstrap be run for: aclocal (GNU automake) 1.16.5 But the Docker image has an older version which triggers the error: aclocal (GNU automake) 1.13.4 Any machine with aclocal 1.16.x works fine. Likely any machine with a version that differs from 1.16.x does not. Running bootstrap fixes that... but also changes the contents of all the Makefile.in files making them appear as a large set of diffs in the repo. This is not an issue inside the docker image since we generally remove the source code after compiling anyway. * Updating the png version of the older flowcharts. * #1815 cleanup * #1815 cleanup * #1851 cleanup * #1851 cleanup * #1815 Added get_att_unixtime * #1851 cleanup * #1815 cleanup * #1815 cleanup * #1815 Support @ for pinterp * #1815 Do not call to_string * #1815 cleanup * Per issue #2081, updated thecontents to match MRMS Version 12.2. Updated existing entries, removed old entries, added new entries. SL (#2199) Co-authored-by: Seth Linden <[email protected]> * #1815 Changed @time to time at plot_data_plane_NCCF_time * Revert "Feature 1815 level value" (#2201) * #1815 Use NcVarInfo members instead API calls * #1815 Added find_var_by_dim_name * #1815 Restored changes for this branch. The changes were reset after revert and merging with latest develop branch * #METplus-Internal_19 cleanup * #METplus-Internal_19 Added more functiofnds and corrected typo * #METplus-Internal_19 Added main.h and main.cc * #METplus-Internal_19 adjusted link order to set oom * #METplus-Internal_19 Renamed main to met_main. Moved oom handler to base API * #METplus-Internal commented out segv_handler which failed to compile at Docker * Fix typo - maintianing * Feature #2204 sum (#2205) * #2202 Added APIs to check the point data (missing data and data types) * #2202 Renamed pointdata_from_xarray to pointdata_from_python_list * #2202 Support python list * #2202 Avoid seg fault by checking point data from python embedding and provides the better log messages * #2202 Support python list * #2202 Added APIs to check the point data (missing data and data types) * #2202 Added APIs to check the point data (missing data and data types) * #2202 Added APIs to check the point data (missing data and data types) * #2202 Added MaskedArray to allowed array data type * Per #2211, enhance warning/error messages when parsing threshold types in dictionary.cc. * #2208 Check existence of ensemble member data before checking bad_data * 2215 Added station_ob to message_type & station_id to station_id for metadata_map * 2215 Add log message for missing message_type and station_id metadata variables. Set NA if a message type is empty string * 2215 Filter out XXX@RecMetaData variables from @metadata variables * Per #2215, update the user's guide to make it consistent with the default ioda2nc config file. * #METplus-Internal_19 Renamed main to met_main and added get_tool_name * Feature sonarqube v11 beta2 (#2222) Co-authored-by: Howard Soh <[email protected]> * Per #2220, update the version number and release notes. * Per #2220, fix typo. * Per #2220, format the METplus-Internal release notes for MET in the same way they were handled for the METplus beta1 release. * Added *hpp * Separated template functions * Separated template functions * Changed argument for get_var_fill_value * Feature 1971 mtd error to warning (#2221) Co-authored-by: Dave Albo <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * #2225 @Level string is not applied as an index and index string without @ is not applied as value * Feature #2220 v11.0.0-beta2_try2 (#2228) * Updating Hera and Jet files for 11.0.0 release * Adding 11.0.0 modulefiles for Hera and Jet * Updating cheyenne installation files * Reformatted slashes in paths for hera config files * Updating installation files for Orion * Removed beta2 references * Adding configuration and modulefiles for gaea installation * Changing METdatadb references to METdataio * Documentation-only commit directly to develop to correct a typo. The default dotsize if 1.0 rather than 10. * Feature 2055 nbm grib2 update (#2234) * Per issue #2055, in read_grib2_record_list() added new PDS numbers 6, 10, 15 (for NBM data). SL * Per issue #2055: in read_grib2_record_list(), in section that validates PDS number changed, for undefined PDS numbers, changed Error to Warning and continue (instead of exit). SL * Per issue #2055, created new entries for NBM grib2 data fields. SL * Per issue #2055. Added more NBM fields based on a 12z, f003 file. SL * Per issue #2055, after interogating several NBM lead-time files, added TMAX field. SL * Per issue #2055, added grib2_nbm.txt to the list. SL * Per #2055, update the grib2_nbm.txt file entries in the table_files Makefiles. * Per issue #2055, fixed syntax entry for TSTM field. SL * Per issue #2055, fixed a few more syntax errors for some of the fields. SL * Per issue #2055, in read_grib2_record_list(), added section to set level values for template number 6 (NBM data). Also added some temporary print statements. SL * Per issue #2055, in read_grib2_record_list(), added code to set levels for table number 10 (like 6). SL * Per #2055, adding support for new GRIB2_perc_val configuration option to filter records based on the forecast percentile value used by GRIB2 product definition templates 6 and 10. Seth, note that I also included the probability filtering logic tweak we discussed, since it was already present in that same file. * Per issue #2055: in find_record_matches(), in section that resolves Prob fields, changed 'break' to 'continue' for non Probs. SL * Per issue #2055, in read_grib2_record_list() cleaned up some commented out code. SL * Per issue #2055, in read_grib2_record_list() cleaned up all print statements. SL * Per issue #2055, added 5 new unit tests for NBM data. SL * Update data2d_grib2.cc * Update data2d_grib2.cc Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 2207 map (#2241) * Feature #2237 tmp_dir (#2242) * Per #2196, updating files for changes to met base image (#2246) * Hotfix to the develop branch to fix typos in the names of 3 tools (grid_stat, gribtab_dat_to_flat, and gsidens2orank). Also removed commented out code from main.cc. * Feature 2250 accum (#2251) * Feature 2198 zenodo (#2252) * Per #2198, adding Seth Linden and Dave Albo to the Author List * Per #2198, updated Dave Albo to David Albo * Per #2198, added three new authors based on submitted AMS abstract * Per #2198, adding .zenodo.json file * Per #2198, adding orcids for all based on Tatiana's suggestion * Update .zenodo.json Co-authored-by: johnhg <[email protected]> * Update .zenodo.json Co-authored-by: johnhg <[email protected]> * Update .zenodo.json Co-authored-by: johnhg <[email protected]> Co-authored-by: johnhg <[email protected]> * Per #1744, adding names for rst files in index.rst * Reword to Grid-Stat opening paragraph * Feature 2196 update dockerfile (#2249) * Per #2196, update for upgrade to Python 3.8.6 * Updated based on new structure of METbaseimage * Changed MET_BASE_IMAGE references to MET_BASE_TAG; Added information to the README.md file * Changed MET_BASE_IMAGE references to MET_BASE_TAG; Updated the version from v1.0 to v1.1 * Update internal/scripts/docker/README.md Based on suggestion from John HG. Co-authored-by: johnhg <[email protected]> * Update internal/scripts/docker/README.md Based on suggestion from John HG. Co-authored-by: johnhg <[email protected]> * Update internal/scripts/docker/README.md Based on suggestion from John HG. Co-authored-by: johnhg <[email protected]> * Update internal/scripts/docker/README.md Based on suggestion from John HG. Co-authored-by: johnhg <[email protected]> * Per #2196, updating README.md based on suggestions from John HG Co-authored-by: johnhg <[email protected]> * Feature 1774 contrib outline (#2253) * creating chapter files * putting a space in to work through the git add and commit process. * Changed formatting of one header Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: jprestop <[email protected]> * Fine tune to grid-stat opening paragraph * Feature 2206 fair crps to ecnt (#2247) * Per issue #2206, for the ECNT line type, added CRPS_EMP_FAIR. SL * Per issue #2206, for ECNT line type, added CRPS_EMP_FAIR. SL * Per issue #2206, for ecnt_columns, added CRPS_EMP_FAIR. SL * Per issue #2206, in write_ecnt_cols(), added crps_emp_fair. SL * Per issue #2206, in ECNTInfo, added crps_emp_fair. SL * Per issue #2206, started stubbing code pieced to calculate crps_emp_fair. SL * Per issue #2206, added new function mean_abs_diff(). SL * Per issue #2206, in compute_pair_vals() updated the code to calculate crps_emp and crps_emp_fair. SL * Per issue #2206, added crps_emp_fair (CRPS_EMP_FAIR) to relevant code pieces. SL * Per issue #2206, added wording for CRPS_EMP_FAIR, also added it to last column of ECNT line type. SL * Per issue #2206, added info for CRPS_EMP_FAIR and also added the math equation for this. SL * Per issue #2206, update the math equation for CRPS_EMP_FAIR. SL * Per issue #2206, added new function weighted_mean_absolute_diff(). SL * Per issue #2206, updated the crps_emp_fair calculation to subtract the weighted_mean_abs_diff. SL * Per issue #2206, in aggr_orank_lines(), updated calculation for crps_emp_fair to use weighted_mean_abs_diff. SL * Per issue #2206, renamed weighted_mean_abs_diff() to wmean_abs_diff(). SL * Per issue #2206, for the crps_emp_fair calculation, changed weighted_mean_abs_diff() to wmean_abs_diff(). SL * Per issue #2206, in aggr_orank_lines(), for the crps_emp_fair calculation, changed weighted_mean_abs_diff() to wmean_abs_diff(). SL * Per issue #2206, updated the math equation for the CRPS_EMP_FAIR calculation. SL * Per issue #2206, fixed a bug in the wmean_abs_diff() function. SL * Per issue #2206, added code for crps_emp_fair. SL * Per issue #2206, in set() function, added checks for crps_emp and crps_emp_fair, for values of 0.0. SL Co-authored-by: Seth Linden <[email protected]> * automated tests: added ability to trigger build of develop-lite docker image that it used in the METplus automated tests * to fix issues with python embedding when the path changes, update development.docker file (apparently unused or rarely previously) to include changes for Python 3.8 including setting the LDFLAG to include -rpath to prevent needing LD_LIBRARY_PATH set to run the tools. Source the env file in the build MET docker script instead of setting the env vars in the configure command * Hotfix to the develop branch to revert the python version from 3.8 back to 3.6. This is done by rolling back the base image in the MET Dockerfiles to dtcenter/met-base:v1.0. Using 3.8 caused the downstream METplus python embedding examples to fail via GHA in the METplus repo. Will write up an issue to investigate more during the MET-11.0.0-beta4 development cycle. * Tweak the build_docker_image.sh script by removing references to MET_BASE_IMAGE and MET_BASE_TAG since those variables are hard-coded in the Dockerfiles themselves and this script includes no logic to handle them. * Revert "Tweak the build_docker_image.sh script by removing references to MET_BASE_IMAGE and MET_BASE_TAG since those variables are hard-coded in the Dockerfiles themselves and this script includes no logic to handle them." This reverts commit 5efc65e. * Hotfix, found 2 more spots where I needed to change v1.1 to v1.0. We should really revisit having this setting spread across so many locations. * Bugfix 2238 link error (#2255) Co-authored-by: Howard Soh <[email protected]> * Feature dtcenter/METplus-Internal#15 add checksum to release (#2254) Co-authored-by: jprestop <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * feature 2216 ioda2nc station_id (#2259) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1942 seeps (#2257) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> * Feature 2142 ascii2nc airnow (#2256) Co-authored-by: Dave Albo <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1954 semilatlon (#2262) Co-authored-by: j-opatz <[email protected]> * Removed future imports since they were used to transition from python 2 to 3. Rearranged logic to be more readible by checking error conditions and exiting if they are not met instead of putting all of the logic to run inside the if statement * Feature 1908 remove ens (#2265) * Feature 2266 v11.0.0-beta3 (#2268) * Minor hotfix for the develop branch. Running bootstrap for develop on seneca changes the order of vx_grid semilatlon slightly. We probably manually editted that file instead of running bootstrap to regenerate it. * Bugfix #2271 develop nbrctc (#2272) Co-authored-by: George McCabe <[email protected]> * Feature 2282 docs (#2283) * #2270 Removed version number for SonrQube scanner pathdiff * Feature #2286 grib_lookup (#2288) * Feature 2227 namespace (#2290) * Per #2227 changes in this commit resolve all errors from removing 'using namespace std' from all header files in the src/basic directory * Attempt to resolve errors in GitHub actions from removing 'using namepsace std' from all src/basic/ libraries, plus removal from src/libcode/vx_gnomon/gnomon.h and src/libcode/vx_nc_util/nc_utils.h * Attempt to fix GHA errors with compilation after removal of 'using namespace std'; locally compiles fine * Per #2227, attempting to fix GHA compilation; Compiltion on seneca works fine * Per #2227, attempt to resolve GHA compilation issues; seneca compilation works fine * Per #2227, attempt to resolve GHA compilation issues; seneca compilation works fine * Per #2227, removing namespace from vx_series_data library * Per #2227, removing namespace std from vx_shapedata library * Per #2227, removing namespace std from vx_data2d_grib2 and vx_data2d_grib libraries * Per #2227, removing namespace std from vx_statistics library * Per #2227, removing namespace std from src/tools/tc_utils * Per #2227, removed namespace std from /src/tools/core header files * Per #2227, clean up and continued work * Per #2227, changes to remove namespace std and netCDF from header files * Per #2227, resolve errors in GitHub Actions build * Per #2227, resolve new errors in GitHub Actions build * Per #2227, resolve new errors in GitHub Actions build * Per #2227, resolve new errors in GitHub Actions build * Per #2227, realign variables * Per #2227, updating comment * Per #2227, updating comment * Per #2227, realign variables * Per #2227, realign variables * Per #2227, realign variables * Per #2227, updating comment * Update src/basic/vx_config/config.tab.cc Co-authored-by: johnhg <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: johnhg <[email protected]> * Feature #2237 test_empty_config (#2291) * Feature #2297 gha (#2299) * GHA hotfix to develop Changes for #2297 included a small typo that I'm fixing directly in the develop branch. * Feature #2297 hotfix to develop A final hotfix to the develop branch to get the build_docker_trigger_metplus.yml workflow working again. I tested on feature_2297_gha branch to confirm that no double-quotes are actually needed. * Update GHA versions * Feature 2281 linker (#2296) Co-authored-by: Julie Prestopnik <[email protected]> * Feature 2036 enhance tc pairs (#2301) Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 2276 enhance ascii2nc to read ndbc buoy data (#2294) Co-authored-by: Dave Albo <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> * Feature 2155 time units months and years (#2300) Co-authored-by: Howard Soh <[email protected]> * removing release-notes out of overview and creating a seperate chapter for this information. (#2304) * Bugfix #2309 develop tcmpr (#2310) * Bugfix #2306 ascii2nc airnow hourly (#2314) Co-authored-by: Dave Albo <[email protected]> * Feature #2058 bias_ratio (#2317) * Feature 2232 refine grid diag (#2316) Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> * Feature #392 tcdiag_line_type (#2315) * Feature #2068 ioda2nc v2.0 (#2307) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> * Feature #2322 v11.0.0-beta4 (#2323) * Feature #2325 ECNT_MAE (#2326) * Hotfix to the develop branch to correct the column numbers in the CNT line type table for the RMSOA, ANOM_CORR_UNCNTR, and SI statistics. * Feature #2332 spread_md (#2333) * Per #2232, add SPREAD_MD to the ECNT line type from Ensemble-Stat and computed for HiRA. * Per #2332, fix compilation error. * Per #2332, update date and release notes for met-11.0.0-beta4. * Per #2332, fix compilation error. Co-authored-by: Howard Soh <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Randy Bullock <[email protected]> Co-authored-by: davidfillmore <[email protected]> Co-authored-by: rgbullock <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: mo-mglover <[email protected]> Co-authored-by: davidalbo <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Molly Smith <[email protected]> Co-authored-by: Dave Albo <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
Currently, stat_analysis ALWAYS writes the output it generates... either to the log output (i.e. printed to the screen and/or copied to a -log file) or to an ascii output file specified by -out. When the -out_stat job command option is provided, it ALSO writes the output to the .stat output file specified. For casual stat_analysis usage which is pretty small, duplicating the output in this way doesn't matter much. But NOAA/WPC is making heavy use of stat_analysis and generating a relatively large amount of .stat output.
This change is, if the job specifies a -out_stat output file, DO NOT write the output in the standard way. Note that this change only applies to jobs which generate true .stat output, including -job aggregate and -job aggregate_stat. But it does not apply to other jobs, like -job summary, -job filter, or -job go_index.
Manually tested the changes. Also compiled code for this PR and ran all of the unit tests on kiowa in:
/d1/projects/MET/MET_pull_requests/met-10.0.0_beta5/feature_1736
As expected, the output from Stat-Analysis has changed, as shown in:
comp_dir_stat_analysis.log
Content that previously was in the -out files is no longer there.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
@j-opatz please review documentation changes (see below) and changes to the logic.
@jprestop please review the code changes and difference in test output.
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Please search for "out_stat" on this page:
https://met.readthedocs.io/en/feature_1736_out_stat/Users_Guide/stat-analysis.html
Do these changes include sufficient testing updates? [Yes]
The existing tests are sufficient since they run a mix of jobs with/without the -out_stat option.
Will this PR result in changes to the test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
The diffs should be limited to the -out files from Stat-Analysis. Whereas previously, they would have always contained all the output, they now only do so if -out_stat was not also used for that job.
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s), Project(s), and Milestone