Feature 1949 cf netcdf documentaton #1951
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
21 tasks
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
| Requirements for CF Compliant NetCDF | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| MET tools uses following attributes and variables for the CF Compliant NetCDF. |
There was a problem hiding this comment.
Suggested change
| MET tools uses following attributes and variables for the CF Compliant NetCDF. | |
| The MET tools use the following attributes and variables for input CF-Compliant NetCDF data. |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 6. (Optional) the "forecast_reference_time" variable for init_time. | ||
|
|
||
| MET processes the CF-Compliant gridded NetCDF files with the projection information. The CF-Compliant NetCDF is defined by the global attribute "Conventions" whose value begins with "CF-" ("CF-<Version_number>"). The global attribute "Conversions" is mandatory. MET accepts the variation of this attribute ("conventions" and "CONVENTIONS"). The value should be started with "CF-" and followed by the version number. MET accepts the attribute value that begins with "CF " ("CF" and a space instead of a hyphen) or "COARDS". |
There was a problem hiding this comment.
Suggested change
| MET processes the CF-Compliant gridded NetCDF files with the projection information. The CF-Compliant NetCDF is defined by the global attribute "Conventions" whose value begins with "CF-" ("CF-<Version_number>"). The global attribute "Conversions" is mandatory. MET accepts the variation of this attribute ("conventions" and "CONVENTIONS"). The value should be started with "CF-" and followed by the version number. MET accepts the attribute value that begins with "CF " ("CF" and a space instead of a hyphen) or "COARDS". | |
| MET processes the CF-Compliant gridded NetCDF files with the projection information. The CF-Compliant NetCDF is defined by the global attribute "Conventions" whose value begins with "CF-" ("CF-<Version_number>"). The global attribute "Conventions" is mandatory. MET accepts the variation of this attribute ("conventions" and "CONVENTIONS"). The value should be started with "CF-" and followed by the version number. MET accepts the attribute value that begins with "CF " ("CF" and a space instead of a hyphen) or "COARDS". |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 1. The global attribute "Conventions". | ||
|
|
||
| 2. The "standard_name" and "units" attributes for coordinate variables. The "axis" attribute ("T" or "time") must exist at the time variable if the "standard_name" attribute does not exist. |
There was a problem hiding this comment.
Suggested change
| 2. The "standard_name" and "units" attributes for coordinate variables. The "axis" attribute ("T" or "time") must exist at the time variable if the "standard_name" attribute does not exist. | |
| 2. The "`standard_name <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#standard-name>`_" and "`units <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#units>`_" attributes for coordinate variables. The "`axis <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#time-axis-ex>`_" attribute ("T" or "time") must exist as the time variable if the "standard_name" attribute does not exist. |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 2. The "standard_name" and "units" attributes for coordinate variables. The "axis" attribute ("T" or "time") must exist at the time variable if the "standard_name" attribute does not exist. | ||
|
|
||
| 3. The "coordinates" attribute at the data variables. It contains the coordinate variable names. |
There was a problem hiding this comment.
Suggested change
| 3. The "coordinates" attribute at the data variables. It contains the coordinate variable names. | |
| 3. The "`coordinates <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#coordinate-types>`_" attribute for the data variables. It contains the coordinate variable names. |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 3. The "coordinates" attribute at the data variables. It contains the coordinate variable names. | ||
|
|
||
| 4. The "grid_mapping" attribute at the data variables for projections and the matching grid mapping variable (optional for the latitude_longitude projection). |
There was a problem hiding this comment.
Suggested change
| 4. The "grid_mapping" attribute at the data variables for projections and the matching grid mapping variable (optional for the latitude_longitude projection). | |
| 4. The "`grid_mapping <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#appendix-grid-mappings>`_" attribute for the data variables for projections and the matching grid mapping variable (optional for the latitude_longitude projection). |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 5. The gridded data should be evenly spaced horizontally and vertically. | ||
|
|
||
| 6. (Optional) the "forecast_reference_time" variable for init_time. |
There was a problem hiding this comment.
Suggested change
| 6. (Optional) the "forecast_reference_time" variable for init_time. | |
| 6. (Optional) the "`forecast_reference_time <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#scalar-coordinate-variables>`_" variable for init_time. |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 2. the lat/lon projection from the "coordinates" attribute from the data variable | ||
|
|
||
| 3. the lat/lon projection from the latitude and longitude variables by tte "standard_name" attribute |
There was a problem hiding this comment.
Suggested change
| 3. the lat/lon projection from the latitude and longitude variables by tte "standard_name" attribute | |
| 3. the lat/lon projection from the latitude and longitude variables by the "standard_name" attribute |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 3. the lat/lon projection from the latitude and longitude variables by tte "standard_name" attribute | ||
|
|
||
| MET is looking for variables with the same name as the dimension and checking the "units" attribute to find the latitude and longitude variables. The valid "units" strings are listed at the table below. MET accepts the variable "tlat" and "tlon" if the dimension names are "nlat" and "nlon”. |
There was a problem hiding this comment.
Suggested change
| MET is looking for variables with the same name as the dimension and checking the "units" attribute to find the latitude and longitude variables. The valid "units" strings are listed at the table below. MET accepts the variable "tlat" and "tlon" if the dimension names are "nlat" and "nlon”. | |
| MET is looking for variables with the same name as the dimension and checking the "units" attribute to find the latitude and longitude variables. The valid "units" strings are listed in the table below. MET accepts the variable "tlat" and "tlon" if the dimension names are "nlat" and "nlon”. |
jprestop
reviewed
Oct 26, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| If there are no latitude and longitude variables from dimensions, MET gets coordinate variable names from the "coordinates" attribute. The matching coordinate variables should have the proper "units" attribute. | ||
|
|
||
| MET gets the time, latitude, and the longitude variables by looking at the standard name: "time", "latitude", and "longitude" as the last option. |
There was a problem hiding this comment.
Suggested change
| MET gets the time, latitude, and the longitude variables by looking at the standard name: "time", "latitude", and "longitude" as the last option. | |
| MET gets the time, latitude, and longitude variables by looking at the standard name: "time", "latitude", and "longitude" as the last option. |
jprestop
requested changes
Oct 26, 2021
There was a problem hiding this comment.
Thank you for writing all of this helpful documentation! I particularly like the examples that you gave for both “Conventions” and “grid_mapping”. I reviewed the documentation and added a few suggestions for modifications, including adding links to the CF conventions documentation for the attributes and variables you mention in the documentation to make it easier for the users to find that information.
With regard to what Marion was looking for (#1949 (comment)):
a) Size of input files (limits, implications etc);
Are there any limitations with regard to size or anything else noteworthy with regard to size that we should include in the documentation?
b) splitting files (e.g. keeping forecast ranges separate, can different variables be in the same file, is it ok to have multi-level data in one file, and implications for how to access this via the format string, do ensemble members need to be in separate files, "detecting"/specifying control members),
Would you be able to address these questions more specifically in the documentation? I did not see these questions addressed, but perhaps they were, because I don’t fully understand all of these questions?
c) filenaming conventions and how MET diagnoses the time/date info and uses it for output file naming.
It looks like how MET diagnoses the time/date info was addressed, however, I did not see information about how MET “uses it for output file naming”. Perhaps that is based on the individual tool being used though?
@JohnHalleyGotway, would you be able to look this over as well, since you are more familiar with the technical details of MET?
|
jprestop
reviewed
Oct 27, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| 1. The global attribute "Conventions". | ||
|
|
||
| 2. The ""`standard_name <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#standard-name>`_" and "`units <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#units>`_" attributes for coordinate variables. The "`axis <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#time-axis-ex>`_" attribute ("T" or "time") must exist as the time variable if the "standard_name" attribute does not exist. |
There was a problem hiding this comment.
Suggested change
| 2. The ""`standard_name <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#standard-name>`_" and "`units <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#units>`_" attributes for coordinate variables. The "`axis <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#time-axis-ex>`_" attribute ("T" or "time") must exist as the time variable if the "standard_name" attribute does not exist. | |
| 2. The "`standard_name <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#standard-name>`_" and "`units <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#units>`_" attributes for coordinate variables. The "`axis <https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#time-axis-ex>`_" attribute ("T" or "time") must exist as the time variable if the "standard_name" attribute does not exist. |
jprestop
reviewed
Oct 27, 2021
met/docs/Users_Guide/data_io.rst
Outdated
| Performance with NetCDF input data | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| There is no limitation on the NetCDF file size. The size of the data variables matters than the file size. NetCDF API loads the meta data first on opening the NetCDF. It's the similar on accesing data variables. It's two API calls: getting the meta data and getting the actual data. The memory is allocated and consumed at the second API call (getting actual data). |
There was a problem hiding this comment.
Suggested change
| There is no limitation on the NetCDF file size. The size of the data variables matters than the file size. NetCDF API loads the meta data first on opening the NetCDF. It's the similar on accesing data variables. It's two API calls: getting the meta data and getting the actual data. The memory is allocated and consumed at the second API call (getting actual data). | |
| There is no limitation on the NetCDF file size. The size of the data variables matters more than the file size. The NetCDF API loads the metadata first upon opening the NetCDF file. It's similar for accessing data variables. There are two API calls: getting the metadata and getting the actual data. The memory is allocated and consumed at the second API call (getting the actual data). |
jprestop
reviewed
Oct 27, 2021
met/docs/Users_Guide/data_io.rst
Outdated
|
|
||
| There is no limitation on the NetCDF file size. The size of the data variables matters than the file size. NetCDF API loads the meta data first on opening the NetCDF. It's the similar on accesing data variables. It's two API calls: getting the meta data and getting the actual data. The memory is allocated and consumed at the second API call (getting actual data). | ||
|
|
||
| The dimensions of the data variables matter. MET requests the NetCDF data need based: 1) 1oading and processing a data plane, and 2) loading and processing the next data plane. This means an extra step for slicing with one more dimension NetCDF input data. The performance is quite different if the compression is enabled with high resolution data. NetCDF does compression per variable. The variables can have different compression levels (0 to 9, 0 is not compressed). The number of decompression is the same between 1 more and 1 less dimension NetCDF input files (combined VS separated). The difference is the amount of data to be decompressed which requires more memory. For example, let's assume the time dimension is 30. NetCDF data with 1 less dimension (no time dimension) does decompression 30 times for nx by ny dataset. NetCDF with 1 more dimension does compression 30 times for 30 by nx by ny dataset & slicing for target time offset. So it's better to have multiple NetCDF files with 1 less dimension than a big file with bigger variable data if compressed. If the compression is not enabled, the file size will be much bigger (more disk space). |
There was a problem hiding this comment.
Suggested change
| The dimensions of the data variables matter. MET requests the NetCDF data need based: 1) 1oading and processing a data plane, and 2) loading and processing the next data plane. This means an extra step for slicing with one more dimension NetCDF input data. The performance is quite different if the compression is enabled with high resolution data. NetCDF does compression per variable. The variables can have different compression levels (0 to 9, 0 is not compressed). The number of decompression is the same between 1 more and 1 less dimension NetCDF input files (combined VS separated). The difference is the amount of data to be decompressed which requires more memory. For example, let's assume the time dimension is 30. NetCDF data with 1 less dimension (no time dimension) does decompression 30 times for nx by ny dataset. NetCDF with 1 more dimension does compression 30 times for 30 by nx by ny dataset & slicing for target time offset. So it's better to have multiple NetCDF files with 1 less dimension than a big file with bigger variable data if compressed. If the compression is not enabled, the file size will be much bigger (more disk space). | |
| The dimensions of the data variables matter. MET requests the NetCDF data needs based on: 1) loading and processing a data plane, and 2) loading and processing the next data plane. This means an extra step for slicing with one more dimension in the NetCDF input data. The performance is quite different if the compression is enabled with high resolution data. NetCDF does compression per variable. The variables can have different compression levels (0 to 9). A value of 0 means no compression, and 9 is the highest level of compression possible. The number for decompression is the same between one more and one less dimension NetCDF input files (combined VS separated). The difference is the amount of data to be decompressed which requires more memory. For example, let's assume the time dimension is 30. NetCDF data with one less dimension (no time dimension) does decompression 30 times for nx by ny dataset. NetCDF with one more dimension does compression 30 times for 30 by nx by ny dataset and slicing for target time offset. So it's better to have multiple NetCDF files with one less dimension than a big file with bigger variable data if compressed. If the compression is not enabled, the file size will be much bigger requiring more disk space. |
|
@hsoh-u Thank you for adding the "Performance with NetCDF input data" section to help address Marion's questions. I made a few other minor suggestions for changes. |
21 tasks
|
@mpm-meto, in an email earlier I said "Marion, I'll wait for feedback from you by Monday early afternoon (Mountain time) before approving the pull request." When I said this, I thought the our beta4 release was scheduled for Tuesday, but it is actually scheduled for Monday. My apologies. If you have time on Monday morning, could you please look this over? I'll plan to go ahead and merge this in to develop on Monday morning (Mountain time) so it can make it into the release. However, please don't hesitate to submit any feedback, even if it's been merged in before you have time to review it and we'll be sure to get that feedback into beta5 if we don't make it into beta4. |
jprestop
approved these changes
Nov 15, 2021
There was a problem hiding this comment.
The changes look good. @TaraJensen has had an opportunity to review this as well and agrees. I approve this PR.
21 tasks
JohnHalleyGotway
added a commit
that referenced
this pull request
Feb 3, 2022
* Per #1906, modify code to pass entire path to make_temp_file_name instead of only the filename so that the function can tell whether or not the file exists. (#1952) Co-authored-by: Julie Prestopnik <[email protected]> * Feature 1761 percent thresh (#1956) * Per issue #1761 in set_perc() adding code to get FBIAS numeric value, like 1.0 or 0.9, etc. SL * Per issue #1761: in set_perc(), modified actual percentile calculation at end to use the extracted FBIAS numeric value (float). SL * Per issue #1761: modified the check on the perc_thresh_freq_bias, just has to be > 0 now. SL * Per issue #1761: cleaned up code in set_perc(). SL * Per #1761, updates to Simple_Node::set_perc() to handle variable frequency bias amounts. Changes include: - Reverting the formatting of this back to how it originally was in the develop branch. In general, just match the formatting of the existing file, so as the minimize the number of difference lines. - Add logic to adjust the percentile to be found based on the requested FBIAS value. Multiplying or dividing the percentile by the FBIAS value depends on the inequality type and whether we're bias adjusting the forecast or observation data. - Adjust the log messages slightly. Please be aware that I'm not totally confident in these changes. They warrant much more testing. This logic is very, very confusing. * Per #1761, call compute_percentile() when double-checking the percentile values. * Per #1761, remove unused variable. * Per #1761, add warning for percentiles > 100. * Per #1761. In set_perc(), after testing cleaned up code. SL * Per issue #1761: adding new config file for testing dynamic FBIAS values. SL * Per issue #1761: added new unit test for dynamic FBIAS values when running grid_stat. SL * Per issue #1761, modified FBIAS section to indicated that the user can use dynamic values that are not 1.0. SL * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Per issue #1761, set nc_pairs_flag = FALSE. SL Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1905 ens_ctrl (#1955) Co-authored-by: j-opatz <[email protected]> * Hotfix after merging changing for #1761. Updating the list of expected output files. * Per #1905, committing a hotfix directly to the develop branch. Reverting the logic for computing the ensemble range back to what it was previously. The new version produced very slight differences in the 6-th or 7-th decimal place when compared to previous results. There's not good reason for these changes which were caused by the order of operations in casting from doubles to floats. Reverting back to the old logic prevents diffs for anyone else downstream and is the prudent choice. * Feature 1957 ascii2nc_python (#1958) * Feature 1949 cf netcdf documentaton (#1951) * #1949 Added CF compliant NetCDF into data IO * #1949 Added commas * #1948 Some corrections for typo and added the links for CF attributes * #1948 Added Performance with NetCDF input data * #1949 Corrected tyoe and applied Juloie's suggestions Co-authored-by: Howard Soh <[email protected]> * Feature 1968 ens_ctrl (#1969) * Feature 1809 gen prob (#1967) * Feature 1970 v10.1.0-beta4 (#1973) * Fix tiny typo in plot_data_plane usage statement. * Hotfix for the develop branch to get met-10.1.0-beta4 compiling on WCOSS. The intel compiler does not allow adjacent >> characters when defining maps. * Updating the beta4 release date from 11/16 to 11/17 since the beta4 release needs to be recreated. * Update the MET PR template to list expected changes up front. * Bugfix 1976 develop cdist (#1979) * #1936 Excludes precip3hr, precip6hr, precip12hr, and precip10min from required variables for MESONET * Per #1985, correcting typo found while creating V10.0.1 in the develop branch as well. * Per #1985, VERY minor change to fix alignment of titles in Point-Stat... updating the develop branch. * #1936 Added an unit test for newer MESONET file * Feature 1989 sort (#1990) * Feature 1991 VCNT (#1992) * Feature 1993 grid_mask (#1994) * #1844 Added vx_pointdata_python * #1844 Added vx_pointdata_python to PYTHON_LIBS & make file for vx_pointdata_python * #1844 Added vx_pointdata_python to SUBDIRS * #1844 Initial release * #1844 Added met_point_data.cc & met_point_data.h * #1844 NcHeaderData is renamed to MetPointHeader and moved to met_point_dataq.h * #1844 NcPointObsData is derived from MetPointObsData. Many methods are moved to the base class MetPointObsData * #1844 Moved varianbles and methods to the base class MetPointData * #1844 override the obs_data pointer to the derived class * #1844 Added pyobject_as_bool & pyobject_as_string_array * #1844 Cleanup include statements and addpointdata_python.h if python is enabled * #18443 Support python embedding * #1844 Initial release * #1844 Make buf_size const * Add GitHub Actions workflow to trigger METplus testing workflow (#2002) * changed names of inputs to send to METplus to match changes to METplus repo * #1965 Excludes the duplicated variable names from multiple input files * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 initialize var_idx * Update the development environment for seneca to add /usr/local/nco/bin and /usr/local/netcdf/bin to the path. Also define MET_TEST_RSCRIPT to point to a new enough version of RSCRIPT that has the ncdf4 package 1.17 or later. * send email address of user who triggered event to METplus workflow as input * Fixup the Rscript diffing logic to support running the nightly build on seneca. The problem here is that the ncdiff -x -v command line options that worked in version 4.7.0 NO LONGER work in 4.9.2. Added a check to skip over the time_bounds variable when processing the ncdiff output. * No real changes here. Just formatting * Feature 1974 message_type_group_map (#1999) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1996 uninitialized variables (#2004) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1695 ensemble single file (gen_ens_prod) (#2001) Co-authored-by: johnhg <[email protected]> * Feature 1965 NB faile with time summary by ioda2nc (#2008) Co-authored-by: Howard Soh <[email protected]> * Feature 1810 shapes (#2005) * #1844 Added log message for use_var_id * #1844 Get use_var_id for python embedding * #1844 Initial release * #1844 Added met_point_obs.py and read_met_point_obs.py * #1844 Added 4 unit test for python embedding of MET point data * More changes to test_util.R to account for differences between 4.7.0 and 4.9.2 of ncdiff. The earlier version reported 0 diffs for the time_bounds, lat, lon, latitude, and longitude variables while the newer version reports their raw values. Simply ignore these variables for now. This isn't a great solution but when we reimplement this testing logic in python, we can address this issue then. * #1844 Added python embedding for point observation data * Feature 1926 gridstat openmp (#1977) Co-authored-by: johnhg <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * feature 1695 ensemble single file (ensemble_stat) (#2007) Co-authored-by: johnhg <[email protected]> * Committing directly to the develop branch. Removing many, many instances of MET_BUILD_BASE and a couple instances of MET_BASE from the examples listed in Appendix A. The inconsistent use of these variables will only cause confusion. Removing them will help avoid that confusion. Big picture... MET_BASE defines the installed 'share/met' directory. It can be used to reference the location of map or polyline files. MET_BUILD_BASE is only used in the context of the test scripts. There, it's defined as the top-level source code directory in which the code was compiled. Its used to reference the location of sample data files or Rscript included in the tarball. I left a couple references to these variables in Appendix A where thier use is not confusing. But I removed all instances which are used to the define the location of the 'bin' directory. Instead, let's just assume the MET binaries are already in their path... or the user knows where to find them. * feature 2011 v10.1.0-beta5 (#2014) * feature 1695 fix issues with ensemble changes (#2012) Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Hotfix for #1695 committed directly to the develop branch. The new Ensemble-Stat config files added for this issue don't define a random number generator seed. This results in different output for each run and prevents the regression test from producing repeatable results. Defining the seed here to fix that. * Adding George to email list for the nightly build. * Fix the links for the met-10.1.0-beta5 release notes. * #1844 Make met_point_obs as abstract class * #1844 correctedb for loop end condition on processing obs bdata * Feature 1546 unit_tests (#2021) * #2020 Added SonarQube related varibales * #2020 Initial release * #2015 Avoid the repeasted debug message if derived varibales are disabled * #1996 Initialize right and left * #1966 Call clear() instead of reset memory for Header variable * #1966 Make sure the levels from variable does not exceed the maxLevel * #1966 Removed unreachable codes * #1966 Do not write into NetcDF if empty data * #2015 Corrected "retained or derived" count * Feature 1546 CI testing (#2029) Co-authored-by: John Halley Gotway <[email protected]> * fixed broken workflow by setting environment variable expected for docker build script * add missing docker tag from push command * added correct docker tag value * fixed incorrect truth data version for updating -ref reference data * Updating develop with changes to the unit tests names. Renaming unit_test.yml to a more generic testing.yml name since it compiles AND/OR runs unit tests. Also make the job names more concise so its easier to see what's running in the action summary window on GitHub. * fixed directory to copy truth data into -- copy command was copying the content of the directory, not including the directory itself * Per #1907, added warning about switch from Ensemble-Stat to Gen-Ens-Prod (#2032) Co-authored-by: Julie Prestopnik <[email protected]> * 2028 Check null for header data * Removed un-reachable code * #2028 Check if the pointer (d) is null * #2015 Changed warning to debug message * #2028 Passing two dimensionl cur and dim array * #2028 Set obs_data * Feature 2003 briercl (#2034) Co-authored-by: jprestop <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: mo-mglover <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Howard Soh <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Feb 12, 2022
* Per #1906, modify code to pass entire path to make_temp_file_name instead of only the filename so that the function can tell whether or not the file exists. (#1952) Co-authored-by: Julie Prestopnik <[email protected]> * Feature 1761 percent thresh (#1956) * Per issue #1761 in set_perc() adding code to get FBIAS numeric value, like 1.0 or 0.9, etc. SL * Per issue #1761: in set_perc(), modified actual percentile calculation at end to use the extracted FBIAS numeric value (float). SL * Per issue #1761: modified the check on the perc_thresh_freq_bias, just has to be > 0 now. SL * Per issue #1761: cleaned up code in set_perc(). SL * Per #1761, updates to Simple_Node::set_perc() to handle variable frequency bias amounts. Changes include: - Reverting the formatting of this back to how it originally was in the develop branch. In general, just match the formatting of the existing file, so as the minimize the number of difference lines. - Add logic to adjust the percentile to be found based on the requested FBIAS value. Multiplying or dividing the percentile by the FBIAS value depends on the inequality type and whether we're bias adjusting the forecast or observation data. - Adjust the log messages slightly. Please be aware that I'm not totally confident in these changes. They warrant much more testing. This logic is very, very confusing. * Per #1761, call compute_percentile() when double-checking the percentile values. * Per #1761, remove unused variable. * Per #1761, add warning for percentiles > 100. * Per #1761. In set_perc(), after testing cleaned up code. SL * Per issue #1761: adding new config file for testing dynamic FBIAS values. SL * Per issue #1761: added new unit test for dynamic FBIAS values when running grid_stat. SL * Per issue #1761, modified FBIAS section to indicated that the user can use dynamic values that are not 1.0. SL * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update met/docs/Users_Guide/config_options.rst Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Update test/config/GridStatConfig_fbias_perc_thresh Co-authored-by: johnhg <[email protected]> * Per issue #1761, set nc_pairs_flag = FALSE. SL Co-authored-by: Seth Linden <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1905 ens_ctrl (#1955) Co-authored-by: j-opatz <[email protected]> * Hotfix after merging changing for #1761. Updating the list of expected output files. * Per #1905, committing a hotfix directly to the develop branch. Reverting the logic for computing the ensemble range back to what it was previously. The new version produced very slight differences in the 6-th or 7-th decimal place when compared to previous results. There's not good reason for these changes which were caused by the order of operations in casting from doubles to floats. Reverting back to the old logic prevents diffs for anyone else downstream and is the prudent choice. * Feature 1957 ascii2nc_python (#1958) * Feature 1949 cf netcdf documentaton (#1951) * #1949 Added CF compliant NetCDF into data IO * #1949 Added commas * #1948 Some corrections for typo and added the links for CF attributes * #1948 Added Performance with NetCDF input data * #1949 Corrected tyoe and applied Juloie's suggestions Co-authored-by: Howard Soh <[email protected]> * Feature 1968 ens_ctrl (#1969) * Feature 1809 gen prob (#1967) * Feature 1970 v10.1.0-beta4 (#1973) * Fix tiny typo in plot_data_plane usage statement. * Hotfix for the develop branch to get met-10.1.0-beta4 compiling on WCOSS. The intel compiler does not allow adjacent >> characters when defining maps. * Updating the beta4 release date from 11/16 to 11/17 since the beta4 release needs to be recreated. * Update the MET PR template to list expected changes up front. * Bugfix 1976 develop cdist (#1979) * #1936 Excludes precip3hr, precip6hr, precip12hr, and precip10min from required variables for MESONET * Per #1985, correcting typo found while creating V10.0.1 in the develop branch as well. * Per #1985, VERY minor change to fix alignment of titles in Point-Stat... updating the develop branch. * #1936 Added an unit test for newer MESONET file * Feature 1989 sort (#1990) * Feature 1991 VCNT (#1992) * Feature 1993 grid_mask (#1994) * #1844 Added vx_pointdata_python * #1844 Added vx_pointdata_python to PYTHON_LIBS & make file for vx_pointdata_python * #1844 Added vx_pointdata_python to SUBDIRS * #1844 Initial release * #1844 Added met_point_data.cc & met_point_data.h * #1844 NcHeaderData is renamed to MetPointHeader and moved to met_point_dataq.h * #1844 NcPointObsData is derived from MetPointObsData. Many methods are moved to the base class MetPointObsData * #1844 Moved varianbles and methods to the base class MetPointData * #1844 override the obs_data pointer to the derived class * #1844 Added pyobject_as_bool & pyobject_as_string_array * #1844 Cleanup include statements and addpointdata_python.h if python is enabled * #18443 Support python embedding * #1844 Initial release * #1844 Make buf_size const * Add GitHub Actions workflow to trigger METplus testing workflow (#2002) * changed names of inputs to send to METplus to match changes to METplus repo * #1965 Excludes the duplicated variable names from multiple input files * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 Processed with the raw variable names and update them when the output is saved. * #1965 Added unit test ioda2nc_same_input * #1965 initialize var_idx * Update the development environment for seneca to add /usr/local/nco/bin and /usr/local/netcdf/bin to the path. Also define MET_TEST_RSCRIPT to point to a new enough version of RSCRIPT that has the ncdf4 package 1.17 or later. * send email address of user who triggered event to METplus workflow as input * Fixup the Rscript diffing logic to support running the nightly build on seneca. The problem here is that the ncdiff -x -v command line options that worked in version 4.7.0 NO LONGER work in 4.9.2. Added a check to skip over the time_bounds variable when processing the ncdiff output. * No real changes here. Just formatting * Feature 1974 message_type_group_map (#1999) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1996 uninitialized variables (#2004) Co-authored-by: Howard Soh <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Feature 1695 ensemble single file (gen_ens_prod) (#2001) Co-authored-by: johnhg <[email protected]> * Feature 1965 NB faile with time summary by ioda2nc (#2008) Co-authored-by: Howard Soh <[email protected]> * Feature 1810 shapes (#2005) * #1844 Added log message for use_var_id * #1844 Get use_var_id for python embedding * #1844 Initial release * #1844 Added met_point_obs.py and read_met_point_obs.py * #1844 Added 4 unit test for python embedding of MET point data * More changes to test_util.R to account for differences between 4.7.0 and 4.9.2 of ncdiff. The earlier version reported 0 diffs for the time_bounds, lat, lon, latitude, and longitude variables while the newer version reports their raw values. Simply ignore these variables for now. This isn't a great solution but when we reimplement this testing logic in python, we can address this issue then. * #1844 Added python embedding for point observation data * Feature 1926 gridstat openmp (#1977) Co-authored-by: johnhg <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * feature 1695 ensemble single file (ensemble_stat) (#2007) Co-authored-by: johnhg <[email protected]> * Committing directly to the develop branch. Removing many, many instances of MET_BUILD_BASE and a couple instances of MET_BASE from the examples listed in Appendix A. The inconsistent use of these variables will only cause confusion. Removing them will help avoid that confusion. Big picture... MET_BASE defines the installed 'share/met' directory. It can be used to reference the location of map or polyline files. MET_BUILD_BASE is only used in the context of the test scripts. There, it's defined as the top-level source code directory in which the code was compiled. Its used to reference the location of sample data files or Rscript included in the tarball. I left a couple references to these variables in Appendix A where thier use is not confusing. But I removed all instances which are used to the define the location of the 'bin' directory. Instead, let's just assume the MET binaries are already in their path... or the user knows where to find them. * feature 2011 v10.1.0-beta5 (#2014) * feature 1695 fix issues with ensemble changes (#2012) Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> * Hotfix for #1695 committed directly to the develop branch. The new Ensemble-Stat config files added for this issue don't define a random number generator seed. This results in different output for each run and prevents the regression test from producing repeatable results. Defining the seed here to fix that. * Adding George to email list for the nightly build. * Fix the links for the met-10.1.0-beta5 release notes. * #1844 Make met_point_obs as abstract class * #1844 correctedb for loop end condition on processing obs bdata * Feature 1546 unit_tests (#2021) * #2020 Added SonarQube related varibales * #2020 Initial release * #2015 Avoid the repeasted debug message if derived varibales are disabled * #1996 Initialize right and left * #1966 Call clear() instead of reset memory for Header variable * #1966 Make sure the levels from variable does not exceed the maxLevel * #1966 Removed unreachable codes * #1966 Do not write into NetcDF if empty data * #2015 Corrected "retained or derived" count * Feature 1546 CI testing (#2029) Co-authored-by: John Halley Gotway <[email protected]> * fixed broken workflow by setting environment variable expected for docker build script * add missing docker tag from push command * added correct docker tag value * fixed incorrect truth data version for updating -ref reference data * Updating develop with changes to the unit tests names. Renaming unit_test.yml to a more generic testing.yml name since it compiles AND/OR runs unit tests. Also make the job names more concise so its easier to see what's running in the action summary window on GitHub. * fixed directory to copy truth data into -- copy command was copying the content of the directory, not including the directory itself * Per #1907, added warning about switch from Ensemble-Stat to Gen-Ens-Prod (#2032) Co-authored-by: Julie Prestopnik <[email protected]> * 2028 Check null for header data * Removed un-reachable code * #2028 Check if the pointer (d) is null * #2015 Changed warning to debug message * #2028 Passing two dimensionl cur and dim array * #2028 Set obs_data * Feature 2003 briercl (#2034) * do not run testing workflow if changes are isolated to the met/docs directory * Per #2013, changed source and README. ci-run-unit (#2039) * Feature 1055 read rot latlon (#2041) * CI: output summary of differences in GHA log to easily see results without having to download the log artifact * #2027 Added a debug message * #2027 ci-run-unit Changed wind thres * #2027 ci-run-unit Increased the precision to debug message * print end of script after error summary, ci-run-unit * #2027 Increased the buffer and cheking the status of sprintf * Feature 1998 standard sections (#2038) * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 missed one * fixing section headers to be consistent with METplus #1998 try again * fixing section headers to be consistent with METplus #1998 I had it right the first time * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 fixing mistake * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section headers to be consistent with METplus #1998 * fixing section header underscore to dash #1998 * updating TOC vs ref 1998 * updating TOC vs ref 1998 completed * trying to fix link #1998 * trying to fix link adding reference back in #1998 * updating TOC again underscore vs dash #1998 * updating TOC again underscore vs dash #1998 * updating TOC vs ref 1998 try * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 take 2 * standardizing TOC sections #1998 take 3 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * re-standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 * updating questions into the TOC. test #1998 * updating questions into the TOC. #1998 * standardizing TOC sections #1998 * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * standardizing TOC sections #1998 fixing typo * cleaning up questions. removing section title from questions. * adding a period #1989. * adding the line breaks back in to match other formatting within MET #1989. * Changed some "^^^" to "----" * Changed "###" to "***" Co-authored-by: jprestop <[email protected]> * Update met_stats.cc #2047 Removed the debug message * Update met_stats.cc Removing variable that was set but never used because SonarQube might complain about that, like Fortify does. * Feature 2040 parse timing (#2048) Co-authored-by: Julie Prestopnik <[email protected]> * CI: Add logic to auto update input data (#2046) * Bugfix 2045 develop hira (#2049) Co-authored-by: jprestop <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: mo-mglover <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: davidalbo <[email protected]> Co-authored-by: lisagoodrich <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
No code changes. Added CF compliant NetCDF into Data IO.
Review documentation: https://met.readthedocs.io/en/feature_1949_cf_netcdf/Users_Guide/data_io.html
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [No]
Will this PR result in changes to the test suite? [No]
If yes, describe the new output and/or changes to the existing output:
Please complete this pull request review by [Fill in date].
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes