Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update GRIB tables in MET based on wgrib2 versions 3.1.4 and 3.4.0 #2780

Closed
8 of 21 tasks
JohnHalleyGotway opened this issue Jan 8, 2024 · 11 comments · Fixed by #3005
Closed
8 of 21 tasks

Update GRIB tables in MET based on wgrib2 versions 3.1.4 and 3.4.0 #2780

JohnHalleyGotway opened this issue Jan 8, 2024 · 11 comments · Fixed by #3005
Assignees
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Library Code priority: high High Priority requestor: NOAA/other NOAA Laboratory, not otherwise specified required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Milestone

Comments

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Jan 8, 2024

Describe the Enhancement

Note that this work was performed using the GRIB2 table input from:

This issue was discussed during the METplus NOAA User Telecon on Jan 8, 2024 (see meeting notes - limited access). Gwen indicated that wgrib2 version 3.1.4 is under development and will include updates to some of the GRIB2 tables. Note that wgrib2 has migrated to GitHub at NOAA-EMC/wgrib2.

NCEP GRIB2 tables have been updated with new WMO-approved variables (27 freezing/melting potential, 28 melt onset date, and 29 freeze onset date in Table 4.2-10-2 and Level 172 in Table 4.5).

A new release v3.1.4 will be installed on Acorn soon for testing. If successful, it will be delivered to WCOSS2.
Use the v3.4.0 release from 8/15/24 instead.

This issue is to:

  • Download the v3.1.4 release when it becomes available.
  • Update the contents of the GRIB2 table files in data/table_files/grib2_*.txt as needed.

Recommend requesting that Gwen (Li-Chuan Chen) serve as the tester for these table updates.

Time Estimate

1 day.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Review default alert labels
  • Select component(s)
  • Select priority
  • Select requestor(s)

Milestone and Projects

  • Select Milestone as the next official version or Backlog of Development Ideas
  • For the next official version, select the MET-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next official version
    Select: MET-X.Y.Z Development project for development toward the next official release
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added type: enhancement Improve something that it is currently doing requestor: NOAA/other NOAA Laboratory, not otherwise specified alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Library Code priority: high High Priority labels Jan 8, 2024
@JohnHalleyGotway JohnHalleyGotway added this to the MET 12.0.0 milestone Jan 8, 2024
@JohnHalleyGotway JohnHalleyGotway self-assigned this Jan 8, 2024
@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Jan 16, 2024

Note that discussion dtcenter/METplus#2465 points out that the FSNOWC variable from GRIB2 table 4.2-0-1 is also missing. Double-check to confirm that updates to this table are included in MET version 12.0.0.

On 10/31/2024, confirmed that FSNOWC is included in this update:

grib2_all.txt:0 1 0 255 0 0 1 121 "FSNOWC" "Fraction of Snow Cover" "Proportion"

@JohnHalleyGotway JohnHalleyGotway added the required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone label May 13, 2024
@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented May 13, 2024

At the METplus NOAA User Telecon on 5/13/24, @MarcelCaron-NOAA requested that grib2_mrms.txt be updated to include ~10 new lines for FRANA for this sample of the 24-hour flat ice product:
MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2.gz

Test to confirm that the ice product data is read from these files well.

@JohnHalleyGotway JohnHalleyGotway changed the title Update GRIB2 tables in MET based on wgrib2 version 3.1.4 Update GRIB2 tables in MET based on wgrib2 version 3.4.0 Oct 23, 2024
@JohnHalleyGotway JohnHalleyGotway moved this from 🟢 Ready to 🎯 Up Next in MET-12.0.0 Development Oct 25, 2024
@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Oct 25, 2024

For this issue, recommend updating additional static data sources:

@JohnHalleyGotway JohnHalleyGotway changed the title Update GRIB2 tables in MET based on wgrib2 version 3.4.0 Update GRIB tables in MET based on wgrib2 version 3.4.0 Oct 30, 2024
@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Oct 30, 2024

Details about updating the GRIB2 tables:

  • wgrib2 version 3.4.0 GRIB2 tables are found in wgrib2/wgrib2/gribtables
  • wgrib2 version 3.1.4 GRIB2 tables are found in: hera:/home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables

Ran the following commands to reformat the existing data files to match the current format in MET:

for file in `find wgrib2 -name "*gribtable.dat"`; do
  echo $file
  echo GRIB2 > NEW/`basename $file`
  cat $file | egrep -i '^ *{' | sed 's/^ *{ *//g' | \
  sed 's/ *, */ /g' | sed 's/ *} *//g' >> \
  NEW/`basename $file`
done

When updating the existing tables, make sure:

  1. There is no embedded whitespace in variable names
  2. There is no embedded whitespace in unit strings (found some in MRMS)
  3. There are 3 strings after each set of integers... variable name, long name, units (long name and units out of order in MRMS)

Had to carefully merge existing grib2_mrms.txt with updated MRMS_gribtable.dat. First confirmed that the updated MRMS_gribtable.dat file does NOT introduce any new table entries by checking the first 8 columns. It just updates variable names, descriptions, and units.

echo new GRIB2 > grib2_mrms_new.txt
while read LINE; do
  hdr=`echo $LINE | cut -d' ' -f1-8`
  if [[ `grep "${hdr} " MRMS_gribtable.dat | wc -l` -eq 1 ]]; then
    grep "${hdr} " MRMS_gribtable.dat >> grib2_mrms_new.txt
  else
    grep "${hdr} " grib2_mrms.txt >> grib2_mrms_new.txt
  fi
done < grib2_mrms.txt

Made no updates to grib2_ndfd.txt since it was only changing empty units strings to ?? which doesn't seem like an improvement.

Wrote up NOAA-EMC/wgrib2#285 to alert them of 2 minor GRIB2 table file issues I encountered.

JohnHalleyGotway added a commit that referenced this issue Oct 30, 2024
…with updates from the 109 defined in the wgrib2 table.
@GwenChen-NOAA
Copy link

GwenChen-NOAA commented Oct 30, 2024 via email

@GwenChen-NOAA
Copy link

GwenChen-NOAA commented Oct 30, 2024 via email

@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Oct 30, 2024

@GwenChen-NOAA, aha, thanks for pointing out my mistake. I had assumed that they'd be listed in order in the gribtable file, but they are not. I'll keep that in mind and be more careful in my checking.

Pulled GRIB tables from hera:/home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables, processed them, and note the following:

  1. No differences found for: BOM_gribtable.dat, KMA_gribtable.dat, NDFD_gribtable.dat, dwd_gribtable.dat, tigge_gribtable.dat, ECMWF_gribtable.dat
  2. Many updates for ncep/gribtable.dat increasing the number of entries to 1669.
  3. Has fewer entries for MRMS_gribtable.dat

@JohnHalleyGotway JohnHalleyGotway changed the title Update GRIB tables in MET based on wgrib2 version 3.4.0 Update GRIB tables in MET based on wgrib2 versions 3.1.4 and 3.4.0 Oct 31, 2024
@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Oct 31, 2024

At the METplus NOAA User Telecon on 5/13/24, @MarcelCaron-NOAA requested that grib2_mrms.txt be updated to include ~10 new lines for FRANA for this sample of the 24-hour flat ice product: MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2.gz

Test to confirm that the ice product data is read from these files well.

@MarcelCaron-NOAA, I can't find a definition for this data. Here's the output from wgrib2:

> wgrib2 MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2 
1:0:d=2024012300:var discipline=209 center=161 local_table=1 parmcat=5 parm=8:500 m above mean sea level:anl:

But no corresponding entry exists in the MRMS GRIB tables:

Can you please advise where the definitions for the FRANA Flat Ice Products can be found?

@MarcelCaron-NOAA
Copy link

@JohnHalleyGotway My understanding is this is a brand new product and I'm not aware of any existing MRMS GRIB table definitions for FRANA ... apart from what I've custom-defined and tested (WCOSS2):
/lfs/h2/emc/vpppg/noscrub/marcel.caron/test/frana/grib2_mrms.txt

 209 10 0 255 161 1 5 4  "FRANA_Flat_60min"  "Freezing Rain Accumulation National Analysis - Flat Ice 1-hour"  "mm"
 209 10 0 255 161 1 5 5  "FRANA_Flat_180min"  "Freezing Rain Accumulation National Analysis - Flat Ice 3-hour"  "mm"
 209 10 0 255 161 1 5 6  "FRANA_Flat_360min"  "Freezing Rain Accumulation National Analysis - Flat Ice 6-hour"  "mm"
 209 10 0 255 161 1 5 7  "FRANA_Flat_720min"  "Freezing Rain Accumulation National Analysis - Flat Ice 12-hour"  "mm"
 209 10 0 255 161 1 5 8  "FRANA_Flat_1440min"  "Freezing Rain Accumulation National Analysis - Flat Ice 24-hour"  "mm"
 209 10 0 255 161 1 5 11  "FRANA_Radial_60min"  "Freezing Rain Accumulation National Analysis - Radial Ice 1-hour"  "mm"
 209 10 0 255 161 1 5 12  "FRANA_Radial_180min"  "Freezing Rain Accumulation National Analysis - Radial Ice 3-hour"  "mm"
 209 10 0 255 161 1 5 13  "FRANA_Radial_360min"  "Freezing Rain Accumulation National Analysis - Radial Ice 6-hour"  "mm"
 209 10 0 255 161 1 5 14  "FRANA_Radial_720min"  "Freezing Rain Accumulation National Analysis - Radial Ice 12-hour"  "mm"
 209 10 0 255 161 1 5 15  "FRANA_Radial_1440min"  "Freezing Rain Accumulation National Analysis - Radial Ice 24-hour"  "mm"

Does that help, or would you need definitions from an operational MRMS GRIB table?

@JohnHalleyGotway
Copy link
Collaborator Author

JohnHalleyGotway commented Oct 31, 2024

Details about updating the GRIB1 tables:

  • GRIB1 tables are embedded in the code of wgrib.c
    • The version number is defined as v1.8.5 (9-2023).
    • Also checked hera:/home/Wesley.Ebisuzaki/sorc/wgrib/wgrib.c but that has an older version v1.8.1.2a, and scanning the diffs, the content really is older.
egrep "^const struct ParmTable" wgrib.c | cut -d'[' -f1 > names.txt
while read NAME; do
  out_file="`echo $NAME | awk '{print $4}' | sed 's/parm_table/grib1/g'`.txt"
  echo "Writing: ${out_file}"
  center=`echo $out_file | cut -d'_' -f2`
  subcenter=`echo $out_file | cut -d'_' -f3 | cut -d'.' -f1`
  if [[ "$center" == "ecmwf" ]]; then
    center_id=98
  elif [[ "$center" == "dwd" ]]; then
    center_id=78
  elif [[ "$center" == "cptec" ]]; then
    center_id=46
  elif [[ "$center" == "ncep" || "$center" == "nceptab" ]]; then
    center_id=7
  elif [[ "$center" == "jra55" ]]; then
    center_id=34
  else
    center_id=$center
  fi
  echo GRIB1 > $out_file
  egrep "$NAME" wgrib.c -A 256 | egrep -v "ParmTable" | tr -d '*' > lines.txt
  while read LINE; do
    n=`echo $LINE | awk '{print $2}'`
    var=`echo $LINE | cut -d'"' -f2`
    desc=`echo $LINE | cut -d'"' -f4 | cut -d'[' -f1 | sed -r 's/ +$//g'`
    units=`echo $LINE | cut -d'"' -f4 | cut -d'[' -f2 | cut -d']' -f1`
    if [[ "${desc}" == "${units}" ]]; then
      units=""
    fi
    echo "${n} ${subcenter} ${center_id} -1 \"${var}\" \"${desc}\" \"${units}\"" >> ${out_file}
  done < lines.txt
done < names.txt

Compare these newly generated files with the existing tables and make any changes that are warranted:

  • DWD Tables (listed in MET as both 78 and 146)
    • Found minor diffs in formatting of units, e.g. m2 versus m**2 - no updates warranted.
    • REMOVED EMBEDDED WHITESPACE OF UNITS FROM MET DWD TABLES.
    • opendiff wgrib2/wgrib/grib1_dwd_002.txt grib1_dwd_2_78.txt
    • opendiff wgrib2/wgrib/grib1_dwd_201.txt grib1_dwd_201_78.txt
    • opendiff wgrib2/wgrib/grib1_dwd_202.txt grib1_dwd_202_78.txt
    • opendiff wgrib2/wgrib/grib1_dwd_203.txt grib1_dwd_203_78.txt
    • opendiff wgrib2/wgrib/grib1_dwd_204.txt grib1_dwd_204_78.txt
    • opendiff wgrib2/wgrib/grib1_dwd_205.txt grib1_dwd_205_78.txt
  • ECMWF Tables
    • opendiff wgrib2/wgrib/grib1_ecmwf_128.txt grib1_ecmwf_128_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_129.txt grib1_ecmwf_129_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_130.txt grib1_ecmwf_130_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_131.txt grib1_ecmwf_131_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_132.txt grib1_ecmwf_132_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_133.txt grib1_ecmwf_133_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_140.txt grib1_ecmwf_140_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_150.txt grib1_ecmwf_150_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_151.txt grib1_ecmwf_151_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_160.txt grib1_ecmwf_160_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_162.txt grib1_ecmwf_162_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_170.txt grib1_ecmwf_170_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_171.txt grib1_ecmwf_171_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_172.txt grib1_ecmwf_172_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_173.txt grib1_ecmwf_173_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_174.txt grib1_ecmwf_174_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_180.txt grib1_ecmwf_180_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_190.txt grib1_ecmwf_190_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_200.txt grib1_ecmwf_200_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_210.txt grib1_ecmwf_210_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_211.txt grib1_ecmwf_211_98.txt
    • opendiff wgrib2/wgrib/grib1_ecmwf_228.txt grib1_ecmwf_228_98.txt
  • JMA Tables:
    • wgrib2/wgrib/grib1_jra55_200.txt
      • DOES NOT EXIST IN MET... add it
  • NCEP Tables:
    • opendiff wgrib2/wgrib/grib1_mdl_nceptab.txt
    • opendiff wgrib2/wgrib/grib1_ncep_opn.txt grib1_ncep_2_7.txt
    • opendiff wgrib2/wgrib/grib1_ncep_reanal.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_128.txt grib1_ncep_128_7.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_129.txt grib1_ncep_129_7.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_130.txt grib1_ncep_130_7.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_131.txt grib1_ncep_131_7.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_133.txt grib1_ncep_133_7.txt
    • opendiff wgrib2/wgrib/grib1_nceptab_140.txt grib1_ncep_140_7.txt

Eliminate all embedded whitespace from units string. Also replace all instanced of ** with ^ to consistently indicate exponentiation in a more human-readable way. For example, replace m**-1s with m/s:

cat grib1_*.txt | cut -d'"' -f6 | egrep ' |\*\*' | sort -u > units.txt
rm units_sed.txt
while read LINE; do
  SEARCH=`echo ${LINE} | sed 's%\*%\\\\*%g'`
  REPLACE=`echo ${LINE} | sed 's%\*\*%^%g'`
  echo sed \'s%\"${SEARCH}\"%\"${REPLACE}\"%g\' \| \\ >> units_sed.txt
done < units.txt

Manually edit units_sed.txt to define the conversions and save it as:
fix_units.sh.gz

@JohnHalleyGotway
Copy link
Collaborator Author

Thanks @MarcelCaron-NOAA! Since those definitions don't conflict with any existing usage, adding them to the grib2_mrms.txt file in my feature branch sounds fine to me, which I did with this commit. I don't know the process for getting them officially accepted and the MRMS GRIB2 table updated, but it may eventually involve submitting a new wgrib2 issue.

JohnHalleyGotway added a commit that referenced this issue Oct 31, 2024
…ng them more human-readable by replacing ** with ^ to indicate exponentiation. Also correct bad units for radiation flux in AF tables by listing them as W/m^2.
@JohnHalleyGotway JohnHalleyGotway moved this from 🎯 Up Next to 🏗 In progress in MET-12.0.0 Development Oct 31, 2024
JohnHalleyGotway added a commit that referenced this issue Nov 1, 2024
…h 'mcg' to represent micrograms. Also replace 'mm6' with 'mm^6' by double-checking against the GRIB1 docs.
JohnHalleyGotway added a commit that referenced this issue Nov 1, 2024
…' for consistent usage across NCEP GRIB tables.
@JohnHalleyGotway JohnHalleyGotway linked a pull request Nov 1, 2024 that will close this issue
17 tasks
@JohnHalleyGotway JohnHalleyGotway moved this from 🏗 In progress to 🔎 In review in MET-12.0.0 Development Nov 1, 2024
JohnHalleyGotway added a commit that referenced this issue Nov 7, 2024
* Per #2780, grib2_bom.txt only has minor change replacing 'Prelim:' with 'Prelim'

* Per #2780, grib2_tigge.txt add new table after eliminating whitespace from units strings

* Per #2780, grib2_all.txt increase the number of entries from 1019 to 1570.

* Per #2780, grib2_kma.txt add new grib2 table file.

* Per #2780, carefully merge the existing 175 MRMS GRIB2 table entries with updates from the 109 defined in the wgrib2 table.

* Per #2780, grib2_ecmwf.txt add new ECMWF GRIB2 table.

* Per #2780, grib2_dwd.txt add new DWD GRIB2 table.

* Per #2870, correct units out of order

* Per #2870, grib2_gpp_af.txt elimiate embedded whitespace from units string to avoid problems writing them to FCST_UNITS and OBS_UNITS output columns

* Per #2780, grib2_ecmwf.txt eliminate whitespace from units string

* Per #2780, add/update GRIB2 tables

* Per #2870, remove duplicate entries I'd added.

* Per #2870, grib2_all.txt update with more entries from this version of wgrib2: hera:/home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables/ncep/gribtable.dat

* Per #2780, grib2_mrms.txt add entries for FRANA as requested by @MarcelCaron-NOAA.

* Per #2780, add new grib1_jra_55_200.txt file that did not previously exist for MET.

* Per #2780, remove embedded whitespace from GRIB1 DWD tables.

* Per #2780, fix all the units by removing embedded whitespace and making them more human-readable by replacing ** with ^ to indicate exponentiation. Also correct bad units for radiation flux in AF tables by listing them as W/m^2.

* Per #2780 eliminate trailing whitespace

* Per #2870, fix typo

* Per #2870, fix typos

* Per #2870, fix typo

* Per #2780, grib1_ncep_129_7.txt fix units by replacing '&micro;g' with 'mcg' to represent micrograms. Also replace 'mm6' with 'mm^6' by double-checking against the GRIB1 docs.

* Per #2780, switch to defining particulate matter units as '10^-6g/m^3' for consistent usage across NCEP GRIB tables.

* Per #2780, update GRIB2 units to show exponentiation correctly by comparing against documentation at https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc

* Per #2780, more tweaks for consistent and corrected units

* Per #2780, update Makefiles for new grib1_jra55_200.txt file

* Per #2870, more tweaks for consistent formatting of units to eliminate the use of parantheses in the denominator. For example, replace 'kg/(m*s)' with 'kg/m/s'.
@github-project-automation github-project-automation bot moved this from 🔎 In review to 🏁 Done in MET-12.0.0 Development Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Library Code priority: high High Priority requestor: NOAA/other NOAA Laboratory, not otherwise specified required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Projects
Status: 🏁 Done
Development

Successfully merging a pull request may close this issue.

3 participants