ILAMB for Cori

[forsyth2]

Support ILAMB on Cori. The Cori component of #229.

[forsyth2]

@chengzhuzhang I get the following channel initialization failed error when using the cfg below. Same issues as in #247 (comment). The error is occurring on https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/ts.bash#L128.

Error:

+ e3sm_to_cmip --output-path /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201/post/atm/180x360_aave/cmip_ts/monthly/tmp_ts_atm_monthly_180x360_aave_1851-1852-0002 --var-list 'pr, tas, rsds, rlds, rsus' --realm atm --input-path /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr --user-metadata /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201/post/scripts/tmp.59944542.5Zje/default_metadata.json --num-proc 12 --tables-path /global/cfs/cdirs/e3sm/diagnostics/cmip6-cmor-tables/Tables
Tue Jun  7 20:38:08 2022: [unset]:_pmi_alps_init:alps_get_placement_info returned with error -1
Tue Jun  7 20:38:08 2022: [unset]:_pmi_init:_pmi_alps_init returned -1
[Tue Jun  7 20:38:08 2022] [c7-5c1s1n1] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(537):
MPID_Init(246).......: channel initialization failed
MPID_Init(647).......:  PMI2 init failed: 1
/var/spool/slurmd/job59944542/slurm_script: line 234: 44767 Aborted                 e3sm_to_cmip --output-path ${dest_cmip}/${tmp_dir} --var-list 'pr, tas, rsds, rlds, rsus' --realm atm --input-path ${input_dir} --user-metadata /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201/post/scripts/${workdir}/default_metadata.json --num-proc 12 --tables-path ${cmortables_dir}
+ '[' 134 '!=' 0 ']'
+ cd /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201/post/scripts
+ echo 'ERROR (3)'
+ exit 3

cfg:

[default]
case = v2.LR.historical_0201
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh"
input = /global/cscratch1/sd/forsyth/ilamb_cori_input/v2.LR.historical_0201
input_subdir = archive/atm/hist
mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
output = /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201
partition = haswell
walltime = "02:00:00"
www = /global/cfs/cdirs/e3sm/www/forsyth/zppy_ilamb_cori_output

[ts]
active = True
frequency = "monthly"
years = "1851:1855:2",
ts_fmt = "cmip"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"

  [[ land_monthly ]]
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"

[ilamb_run]
active = True
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1851:1853:2",

[forsyth2]

channel initialization failed error

Using srun fixes this error.

[forsyth2]

Not going to commit the example cfg, but copying here for reference:

[default]
case = v2.LR.historical_0201
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh"
input = /global/cscratch1/sd/forsyth/ilamb_cori_input/v2.LR.historical_0201
input_subdir = archive/atm/hist
mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
output = /global/cscratch1/sd/forsyth/zppy_ilamb_cori_output/v2.LR.historical_0201
partition = haswell
qos = debug
walltime = "00:30:00"
www = /global/cfs/cdirs/e3sm/www/forsyth/zppy_ilamb_cori_output

[ts]
active = True
frequency = "monthly"
years = "1851:1855:2",
ts_fmt = "cmip"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"

  [[ land_monthly ]]
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"

[ilamb_run]
active = True
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1851:1853:2",