Error in writing land model output for e3smv2 run

[rebassoo]

I am running e3smv2 with the main-2.0 branch (date 20231227) and running into issues writing the land model output which kills the job on perlmutter. I have successfully run from 1970-2014 with Compset F2010 and a time-varying sst using the model_start_type="initial". However, when I try to use the end of that run as the Initial Conditions for a new 45 year run starting in 1970 I have issues. I run with model start type "hybrid" and specify all the proper info for the reference case. It runs for 4 years fine (STOP_N="4") but when it tries to resubmit it essentially crashes very quickly after the resubmission when trying to write the land model .h0. file with the error at the end of this post. I have tried changing the BFBFLAG to false to see if somehow it is some numerical instability that can be avoided, but that didn't work.

If there is a different forum I should post this to please let me know. Thanks!

256: PIO: FATAL ERROR: Aborting... An error occured, Writing variables (number of variables = 180) to file (./20231227.v2.LR.F2010-FAMIP-hybrid-01-19-45year-M.elm.h0.1973-01.nc, ncid=149) using PIO_IOTYPE_PNETCDF iotype failed. Non blocking write for variable (TWS_MONTH_BEGIN, varid=205) failed (Number of subarray requests/regions=1, Size of data local to this process = 5400). NetCDF: Numeric conversion not representable (err=-60). Aborting since the error handler was set to PIO_INTERNAL_ERROR... (/global/u2/r/rebassoo/E3SM/fme/code/20231227-maint20/externals/scorpio/src/clib/pio_darray_int.c: 395)

[ndkeen]

Similar issue here #6195 with potential fix

[rljacob]

Sounds like you are running in to an issue that we just posted a fix for in #6206

[milenaveneziani]

Note that the fix won't help when starting a restart from a previous run that didn't have the fix. Because the 'fixed' model will look for a variable in the lnd restart file that was not saved in the previous run. So, the way to go is:

1. add the fix
2. add a dummy variable to the last lnd restart file of the run you are branching off of.
   I'll let @beharrop comment more on point 2) since he found the solution to that.

[beharrop]

As @milenaveneziani mentioned, land restart files written prior to the fix in #6206 will be missing variable ENDWB, so trying to restart with such a file will crash when that variable isn't found. One solution is to write variable ENDWB with zeros to the file. The below script works for adding in ENDWB to the land restart (including adding in the appropriate metadata).

#!/bin/bash

infile=$1
tempfile=temp.elm.r.nc
outfile=$1.withENDWB

ncap2 -s 'ENDWB=0.*H2OSFC' $infile $tempfile
ncap2 -s 'defdim("levurb",5)' $tempfile $outfile

rm $tempfile

ncatted -O -a long_name,ENDWB,o,c,'water balance at end of timestep' $outfile
ncatted -O -a units,ENDWB,o,c,'kg/m2' $outfile
ncatted -O -a interpinic_flag,ENDWB,o,i,1 $outfile
ncatted -O -a interpinic_flag_meanings,ENDWB,o,c,'1=nearest neighbor, 2=copy directly, 3=skip' $outfile
ncatted -O -a _FillValue,ENDWB,o,d,1.e+36 $outfile
ncatted -O -a missing_value,ENDWB,o,d,1.e+36 $outfile

[rebassoo]

Ok, thanks for pointing out the fix as well as the details about needing to start the run new.