ERROR in VAR field settings in ensemble_stat conf file

[BinbinZhou-NOAA]

I tried to run METplus EnsembleStat to verify PRMSL (sea level pressure) against prepbufr.
The VAR setting for PRMSL is

ENS_VAR1_NAME = PRMSL
ENS_VAR1_LEVELS = L0
FCST_VAR1_NAME = {ENS_VAR1_NAME}
OBS_VAR1_NAME = PMO
BOTH_VAR1_LEVELS = {ENS_VAR1_LEVELS}

If just this one field, running is OK, but if put this with some other fields (several VARn), then running got error.
The error message says:

ERROR :
ERROR : NumArray::operator[] (int) const -> range check error ... Length = 0, i = 0
ERROR :

What does this error mean?

Binbin

[georgemccabe]

Hi @BinbinZhou-NOAA,
That error is coming from the ensemble_stat C++ application log output. It looks like it is trying to parse an array that requires at least 1 value but it is empty. It is difficult to tell why this is happening with this context. Could you please send us your METplus final conf file and the full log file that contains this error?

[BinbinZhou-NOAA]

George,

I have copied the running directory to HERA. If you can access HERA, please go to directory

/scratch2/NCEPDEV/fv3-cam/noscrub/Binbin.Zhou/evs_href_grid2obs_dev.208228/grid2obs/run_href_CONUS.1fhr_system
You can see the final conf file final_href_g2o_CONUS_1fhr_sfc.conf
The log file is in /log directory

Binbin

[BinbinZhou-NOAA]

Since only PRMSL has such an issue. The problem may be in the prepbufr nc file. In generating the prepbufr NC file,
This setting is used in the PB2NC conf file:
PB2NC_MET_CONFIG_OVERRIDES = obs_prefbufr_map = [ { key = "ZOB"; val = "HGT"; }, ..... { key = "PMO"; val = "PRMSL"; } ... ]
But I can not see the PRMSL in the output prepbufr's NC file, still PMO. So has setting OBS_VAR1_NAME = PMO and BOTH_VAR1_LEVELS = L0
This is just a hint.

[georgemccabe]

In MET version 10.1.0 the variable obs_prefbufr_map was renamed to obs_prepbufr_map to fix a typo (pref to prep). Typically obs_bufr_map is set by users and obs_prepbufr_map is not modified, so I would recommend removing obs_prefbufr_map from PB2NC_MET_CONFIG_OVERRIDES and instead set:

PB2NC_OBS_BUFR_MAP = { key = "ZOB"; val = "HGT"; }, ..... { key = "PMO"; val = "PRMSL"; } ...

That should change the PB2NC output field to PRMSL so you can change the OBS_VAR1_NAME back to PRMSL.

It is hard to tell what exactly is causing the MET ERROR in your log file. You could raise the verbosity of EnsembleStat with LOG_ENSEMBLE_STAT_VERBOSITY and see if any useful hints are found in the log right above the error you see. If not, you could copy all of the input files to Hera and I can take a closer look.

I'm not sure if this is relevant, but I did notice some warnings that PRMSL was not found in 3 of the m03 and m04 ensemble inputs:

WARNING: process_n_vld() -> ensemble field "PRMSL/L0" not found in file "/lfs/h2/emc/vpppg/noscrub/binbin.zhou/com/evs/v1.0/prep/cam/hr
ef/href.20220717/href.m03.t18z.conus.f06"
...
WARNING: process_n_vld() -> ensemble field "PRMSL/L0" not found in file "/lfs/h2/emc/vpppg/noscrub/binbin.zhou/com/evs/v1.0/prep/cam/hr
ef/href.20220717/href.m04.t18z.conus.f06"

[BinbinZhou-NOAA]

George,

I have corrected the setting in PB2NC conf file and rerun the METplus PB2NC. Now the PB2NC output NC file contains PRMSL instead of PMO.
But running METplus EnsembleStat with the new PB2NC file, the error is still as before in the log file
ERROR :
ERROR : NumArray::operator[] (int) const -> range check error ... Length = 0, i = 0
ERROR :
Even though I raised VERBOSITY to 5, the log file can not provide further detailed info about this error.

So now I select one cycle of all verification data, including 10 href member files, prepbufr NC file, mask file, conf file, and put them onto HERA. The path is
/scratch2/NCEPDEV/fv3-cam/noscrub/Binbin.Zhou/g2o_href_prmsl
In this directory, you can see /log, /stat, /stage, and /tmp sub-directories. These are copied from my run on WCOSS, not run on HERA.
You can also see the final conf. This also run from WCOSS.
The input files are in /input_href.20220718 which includes 10 members and prepbufr nc file
The conf file is in /conf
The mask file is in /mask
To help you to test on HERA, I created an script file run_grid2obs_href.sh which defines all required parameters used by final conf
and the conf file in /conf. You can source this script before you test
The Ensemble_stat verification is to verify 6hr HREF ensemble forecast (cycled at 06Z, 20220718) validated at 12Z, 20220718
Hope you you can test this small case to see if you can repeat this error or not.
If something is missing for this case, please let me know.

Thanks!

Binbin

[georgemccabe]

Hi @BinbinZhou-NOAA, thank you for making those files available. I was able to set up, run the use case, and replicate the error. I will let you know what I find.

[BinbinZhou-NOAA]

You can see that in /stat/HREF, ensemble*.nc can be generated, but no stat file. This implies input member files are fine.
So the problem may be in the obs file

[georgemccabe]

I looked into this error with @hsoh-u. We discovered that the error/crash is happening because 2 of the ensemble files do not have the PRMSL/L0 field. This is noted in the log file as warnings (see previous message in this discussion). It looks like there is a bug in the code that causes the crash instead of skipping the missing ensembles when computing the pair values. It appears to properly handle the situation when only PRMSL is provided. @hsoh-u is creating a GitHub issue to handle this situation. One of us will post a link to the issue when it is created.

In the meantime, I think you should be able to run the use case separately for each field and get the desired results.

[BinbinZhou-NOAA]

Yes, I already knew href.m03 and href.m04 have no PRMSL field. So I set following 2 in the conf,
One if for member missing, another for field missing:
ENSEMBLE_STAT_ENS_THRESH = 0.4
ENSEMBLE_STAT_ENS_VLD_THRESH = 0.4

The strange is why running with only PRMSL (that is, set VAR1 to PRMSL and remove all other fields) in the conf
can successful? In this case, the bug is not a bug?

[hsoh-u]

ensemble_stat works if the order of variables is switched (PRMSL first and VIS next). I think this (two variables have different missing members) is a special case and ensemble_stat needs improvement on handling the missing members.

[hsoh-u]

I'm sorry that the changed order does not work. I re-run the ensemble_stat with changed order, and it was not successful.

[BinbinZhou-NOAA]

I ran the case of switching the order of VAR1 and VAR2, but still failed in same error

[georgemccabe]

This work can be tracked in dtcenter/MET#2208

[georgemccabe]

This work was completed.

[BinbinZhou-NOAA]

Sure, I will trace the case. Thanks! Binbin

…

On Tue, Jul 26, 2022 at 3:16 PM George McCabe ***@***.***> wrote: This work can be tracked in dtcenter/MET#2208 <dtcenter/MET#2208> — Reply to this email directly, view it on GitHub <#1700 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFKMQAJA7KBNQJVIXVTVWA2P7ANCNFSM535EHLTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683