-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Add verification of snowfall accumulation #853
Changes from 21 commits
cb3a355
338dbdf
6ea427b
94fabfd
412dd33
6385801
53ad792
cb2fe28
1d80d7b
b7c3f2e
951137c
dda8aea
c64a051
e35769f
8409c61
fc1d90b
25446be
edf6e3e
e6a906a
a23b55e
12fc095
6557a3f
666fbe9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -151,23 +151,31 @@ METplus Parameters | |
* ``SS`` refers to the two-digit valid seconds of the hour | ||
|
||
``CCPA_OBS_DIR``: (Default: "") | ||
User-specified location of top-level directory where CCPA hourly precipitation files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_CCPA`` task. (This task is activated in the workflow by using the taskgroup file ``parm/wflow/verify.yaml``). | ||
User-specified location of top-level directory where CCPA hourly precipitation files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_CCPA`` task. (This task is activated in the workflow by using the taskgroup file ``parm/wflow/verify_pre.yaml``). | ||
|
||
METplus configuration files require the use of a predetermined directory structure and file names. If the CCPA files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2``, where YYYYMMDD and HH are as described in the note :ref:`above <METParamNote>`. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``CCPA_OBS_DIR`` directory. This path must be defind as ``/<full-path-to-obs>/ccpa/proc``. METplus is configured to verify 01-, 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. | ||
|
||
.. note:: | ||
There is a problem with the valid time in the metadata for files valid from 19 - 00 UTC (i.e., files under the "00" directory). The script to pull the CCPA data from the NOAA HPSS (``scripts/exregional_get_obs_ccpa.sh``) has an example of how to account for this and organize the data into a more intuitive format. When a fix is provided, it will be accounted for in the ``exregional_get_obs_ccpa.sh`` script. | ||
|
||
``NOHRSC_OBS_DIR``: (Default: "") | ||
User-specified location of top-level directory where NOHRSC 06- and 24-hour snowfall accumulation files (available every 6 and 12 hours respectively) used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NOHRSC`` task. (This task is activated in the workflow by using the taskgroup file ``parm/wflow/verify_pre.yaml``). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Change the task name to lowercase, i.e. |
||
|
||
METplus configuration files require the use of a predetermined directory structure and file names. If the NOHRSC files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/sfav2_CONUS_{AA}h_{YYYYMMDD}{HH}_grid184.grb2``, where AA is the 2-digit accumulation duration, and YYYYMMDD and HH are as described in the note :ref:`above <METParamNote>`. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``NOHRSC_OBS_DIR`` directory. This path must be defind as ``/<full-path-to-obs>/nohrsc/proc``. METplus is configured to verify 06-, and 24-h accumulated precipitation using NOHRSC files. | ||
|
||
.. note:: | ||
Due to limited availability of NOHRSC observation data on NOAA HPSS, and the likelihood that snowfall acumulation verification will not be desired outside of winter cases, this verification option is currently not present in the workflow by default. In order to use it, the verification environment variable VX_FIELDS should be updated to include ``ASNOW``. This will allow the related workflow tasks to be run. | ||
|
||
``MRMS_OBS_DIR``: (Default: "") | ||
User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_MRMS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``/<full-path-to-obs>/mrms/proc``. | ||
User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_MRMS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``/<full-path-to-obs>/mrms/proc``. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Change the task name to lowercase, i.e. |
||
|
||
METplus configuration files require the use of a predetermined directory structure and file names. Therefore, if the MRMS files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2``, where YYYYMMDD and {HH}{mm}{SS} are as described in the note :ref:`above <METParamNote>`. | ||
|
||
.. note:: | ||
METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified; since MRMS composite reflectivity files do not always exactly match the valid time, a script (within the main script that retrieves MRMS data from the NOAA HPSS) is used to identify and rename the MRMS composite reflectivity file to match the valid time of the forecast. The script to pull the MRMS data from the NOAA HPSS has an example of the expected file-naming structure: ``scripts/exregional_get_obs_mrms.sh``. This script calls the script used to identify the MRMS file closest to the valid time: ``ush/mrms_pull_topofhour.py``. | ||
|
||
``NDAS_OBS_DIR``: (Default: "") | ||
User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``/<full-path-to-obs>/ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. | ||
User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``/<full-path-to-obs>/ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Change the task name to lowercase, i.e. |
||
|
||
METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above <METParamNote>`. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_obs_ndas.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -170,7 +170,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | |
| | PARTITION_HPSS, QUEUE_HPSS, PARTITION_FCST, QUEUE_FCST, | | ||
| | RUN_CMD_UTILS, RUN_CMD_FCST, RUN_CMD_POST, SLURM_NATIVE_CMD, | | ||
| | MODEL, MET_INSTALL_DIR, METPLUS_PATH, MET_BIN_EXEC, CCPA_OBS_DIR, | | ||
| | MRMS_OBS_DIR, NDAS_OBS_DIR | | ||
| | MRMS_OBS_DIR, NDAS_OBS_DIR, NOHRSC_OBS_DIR | | ||
+-----------------------------+-----------------------------------------------------------------------+ | ||
| Workflow | WORKFLOW_ID, USE_CRON_TO_RELAUNCH, CRON_RELAUNCH_INTVL_MNTS, | | ||
| | EXPT_BASEDIR, EXPT_SUBDIR, EXEC_SUBDIR, DOT_OR_USCORE, | | ||
|
@@ -278,6 +278,8 @@ The user must specify certain basic experiment configuration information in a `` | |
+--------------------------------+-------------------+------------------------------------+ | ||
| CCPA_OBS_DIR | "" | "" | | ||
+--------------------------------+-------------------+------------------------------------+ | ||
| NOHRSC_OBS_DIR | "" | "" | | ||
+--------------------------------+-------------------+------------------------------------+ | ||
| MRMS_OBS_DIR | "" | "" | | ||
+--------------------------------+-------------------+------------------------------------+ | ||
| NDAS_OBS_DIR | "" | "" | | ||
|
@@ -722,12 +724,14 @@ Users who have already staged the observation data needed for METplus (i.e., the | |
|
||
platform: | ||
CCPA_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ccpa/proc | ||
NOHRSC_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/nohrsc/proc | ||
MRMS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/mrms/proc | ||
NDAS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ndas/proc | ||
rocoto: | ||
tasks: | ||
taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' | ||
task_get_obs_ccpa: | ||
task_get_obs_nohrsc: | ||
task_get_obs_mrms: | ||
task_get_obs_ndas: | ||
|
||
|
@@ -827,18 +831,24 @@ In addition to the baseline tasks described in :numref:`Table %s <WorkflowTasksT | |
| **Workflow Task** | **Task Description** | | ||
+=======================+============================================================+ | ||
| GET_OBS_CCPA | Retrieves and organizes hourly :term:`CCPA` data from NOAA | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @willmayfield @gspetro This is not directly related to this PR, but we will have to update the names of the vx and related tasks in this table, e.g. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @gsketefian I'm actually working on updates now! The code is in my text/ug-updates branch. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One thing I was trying to decide was whether to name the (meta)tasks based on what is in the YAML config file or based on what the Rocoto/log output is. I went with what is in the YAML because that is the name users may actually add/change/remove in their config file. |
||
| | HPSS. Can only be run if ``verify.yaml`` is included in a | | ||
| | ``tasksgroups`` list *and* user has access to NOAA | | ||
| | HPSS. Can only be run if ``verify_pre.yaml`` is included | | ||
| | in a ``tasksgroups`` list *and* user has access to NOAA | | ||
| | :term:`HPSS` data. | | ||
+-----------------------+------------------------------------------------------------+ | ||
| GET_OBS_NOHRSC | Retrieves and organizes hourly :term:`NOHRSC` data from | | ||
| | NOAA HPSS. Can only be run if ``verify_pre.yaml`` is | | ||
| | included in a ``tasksgroups`` list *and* user has access | | ||
| | to NOAA :term:`HPSS` data. ``ASNOW`` should also be added | | ||
| | to the ``VX_FIELDS`` list. | | ||
+-----------------------+------------------------------------------------------------+ | ||
| GET_OBS_NDAS | Retrieves and organizes hourly :term:`NDAS` data from NOAA | | ||
| | HPSS. Can only be run if ``verify.yaml`` is included in a | | ||
| | ``tasksgroups`` list *and* user has access to NOAA | | ||
| | HPSS. Can only be run if ``verify_pre.yaml`` is included | | ||
| | in a ``tasksgroups`` list *and* user has access to NOAA | | ||
| | :term:`HPSS` data. | | ||
+-----------------------+------------------------------------------------------------+ | ||
| GET_OBS_MRMS | Retrieves and organizes hourly :term:`MRMS` composite | | ||
| | reflectivity and :term:`echo top` data from NOAA HPSS. Can | | ||
| | only be run if ``verify.yaml`` is included in a | | ||
| | only be run if ``verify_pre.yaml`` is included in a | | ||
| | ``tasksgroups`` list *and* user has access to NOAA | | ||
| | :term:`HPSS` data. | | ||
+-----------------------+------------------------------------------------------------+ | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
#!/bin/bash | ||
|
||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# This script pulls NOHRSC observation data for comparison to the model for | ||
# the requested accumulations. Supported accumulations: 01h, 03h, and 06h. | ||
# NOTE: Accumulation is currently hardcoded to 06h. | ||
# The verification uses MET/pcp-combine to sum 06h files into | ||
# desired accumulations. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
|
||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Source the variable definitions file and the bash utility functions. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
. $USHdir/source_util_funcs.sh | ||
source_config_for_task " " ${GLOBAL_VAR_DEFNS_FP} | ||
. $USHdir/job_preamble.sh | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Save current shell options (in a global array). Then set new options | ||
# for this script/function. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
{ save_shell_opts; . $USHdir/preamble.sh; } > /dev/null 2>&1 | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Get the full path to the file in which this script/function is located | ||
# (scrfunc_fp), the name of that file (scrfunc_fn), and the directory in | ||
# which the file is located (scrfunc_dir). | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" ) | ||
scrfunc_fn=$( basename "${scrfunc_fp}" ) | ||
scrfunc_dir=$( dirname "${scrfunc_fp}" ) | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Print message indicating entry into script. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
print_info_msg " | ||
======================================================================== | ||
Entering script: \"${scrfunc_fn}\" | ||
In directory: \"${scrfunc_dir}\" | ||
|
||
This is the J-job script for the task that pulls NOHRSC observation data | ||
for verification purposes. | ||
========================================================================" | ||
|
||
# | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Call the ex-script for this J-job and pass to it the necessary varia- | ||
# bles. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
$SCRIPTSdir/exregional_get_obs_nohrsc.sh || \ | ||
print_err_msg_exit "\ | ||
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed." | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Run job postamble. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
job_postamble | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
# Restore the shell options saved at the beginning of this script/func- | ||
# tion. | ||
# | ||
#----------------------------------------------------------------------- | ||
# | ||
{ restore_shell_opts; } > /dev/null 2>&1 | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change the task name to lowercase, i.e.
GET_OBS_CCPA
-->get_obs_ccpa
.