global time series failed with index out of bounds

[jli628]

Hi @xylar ,

I'm Jianfeng Li from PNNL. I am running zppy for an 11-year coupled simulation. However, I encountered the below error for the global time series diagnostic (other diagnostics ran successfully). Does anyone have any suggestions about it? Is it because the simulation is too short (only 11 years)? Thanks.

Setting reference time units to days since 1986-01-01 00:00:00
Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.8.1_pm-cpu/bin/cdscan", line 1843, in
main(sys.argv)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.8.1_pm-cpu/bin/cdscan", line 1309, in main
vartime[0],
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.8.1_pm-cpu/lib/python3.10/site-packages/cdms2/axis.py", line 2212, in getitem
raise IndexError('index out of bounds')
IndexError: index out of bounds

zppy_config_for_B_compset_v2_1_LR_historical.cfg.txt

[xylar]

@forsyth2 or @golaz, can you take a look at this.

@jli628, I don't have much expertise with zppy.

[forsyth2]

@jli628 I'm running my own version of your cfg, but it usually takes a while to get compute nodes on Perlmutter. Can you open up permissions on /pscratch/sd/j/jli628/E3SMv3atm/zppy_climo_B_compset, so I can look at your output files?

[jli628]

@forsyth2 Thank you for testing it. Perlmutter was unstable yesterday evening. I changed the permission of the whole directory of /pscratch/sd/j/jli628/E3SMv3atm. Could you please try it again? Thank you for your time and help.

[jli628]

@forsyth2 I have another question regarding the cfg variables. For the below variables, can I set ref_start_yr=1986 and ref_final_yr=1996 to match the model simulation period? Will it produce the same results as the current setting? Thank you!

[e3sm_diags]
active = True
years = "1986:1996:11",
ts_num_years = 11
ref_start_yr = 1979
ref_final_yr = 2016
walltime = "08:00:00"
qos = "regular"

[forsyth2]

I changed the permission of the whole directory of /pscratch/sd/j/jli628/E3SMv3atm.

I'm looking at my own completed run and the logs in your directory. The only thing I can think of is that it doesn't like the 11-year increment, but I was fairly certain that was resolved a while ago (#208). I also tried running a version of your cfg on 10 years but the ts task failed. I'll see if I can find anything else.

can I set ref_start_yr=1986 and ref_final_yr=1996 to match the model simulation period?

In many of the diagnostics sets for E3SM diags, the top panel is test, the middle panel is ref, and the bottom panel is diff. I believe you should be able to set the ref parameters accordingly based on what data is available for whatever you have chosen as ref (in your case /global/cfs/cdirs/e3sm/diagnostics/observations/Atm/climatology/).

[jli628]

Thank you, @forsyth2 , for the explanation. I understand the meanings of ref_start_yr and ref_final_yr now.

[forsyth2]

$ cd /pscratch/sd/j/jli628/E3SMv3atm/zppy_climo_B_compset/post/scripts

# Check for errors
$ grep -v "OK" *status
global_time_series_1986-1996.status:ERROR (3)

$ emacs global_time_series_1986-1996.bash
# Looking in this file, `ERROR (3)` is the exit status if `cdscan -x glb.xml mpaso.glb*.nc` fails in `${case_dir}/post/ocn/glb/ts/monthly/${ts_num_years}yr`

$ cd ../ocn/glb/ts/monthly/11yr/
$ ls
mpaso.glb.198601-199612.nc

# Load Unified
$ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

# Look at contents
$ ncdump -h mpaso.glb.198601-199612.nc
netcdf mpaso.glb.198601-199612 {
dimensions:
	time = UNLIMITED ; // (0 currently)
	nbnd = 2 ;
variables:
	double time(time) ;
		time:long_name = "time" ;
		time:units = "days since 1986-01-01 00:00:00" ;
		time:calendar = "noleap" ;
		time:bounds = "time_bnds" ;
	double time_bnds(time, nbnd) ;
		time_bnds:long_name = "time interval endpoints" ;
	double ohc(time) ;
		ohc:long_name = "total ocean heat content" ;
		ohc:units = "J" ;
	double volume(time) ;
		volume:long_name = "sum of the volumeCell variable over the full domain, used to normalize global statistics" ;
		volume:units = "m^3" ;
}

Notice time = UNLIMITED ; // (0 currently). This error also occurred in #391: #391 (reply in thread). This could be because short-term archiving wasn't done first.