Slow, potentially serial reading of netcdf files by E3SM

[ndkeen]

For a v2 ne30 F case, I was noticing that every 15th day was much slower than the other days. Digging further, I discovered it was because of a file read. This particular file:

/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc/CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc

Looking at the spiostats, I see that a ne30 case ran for 1 year, time spent reading this file was reported as 216 seconds.
I suspect it's being read every 15th day of the month. Which is about 18s per month. I see the 15th day of each month as about 6x more expensive than other days. The file type is classic and it's not very large file -- so I might assume its being read in serial?

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/20240812.v2.LR.F2010-FAMIP-hybrid-03-12-45year

[rljacob]

Related: #6670

[ndkeen]

Updates here.

First, I wanted to try this same launch script with current master. It does run (though I needed to remove 1 specific output), but for whatever reason, does not read the same problematic volc file noted above. Surely good reason for that. But as I'm just trying to reproduce what I saw before, I checked out maint-2.0 again and reran. Sure enough, same issue

After 1 year, spent 627 seconds reading from
/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc/CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc

Then I changed the format of this file from classic to cdf5 and tried again. This time it spent 386 seconds reading. So while almost 2x faster, it's still clearly doing something not quite right as the file itself is small. I'm just going to leave that file in cfd5 format on NERSC unless I hear otherwise.

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/test-20240812.v2.LR.F2010-FAMIP-hybrid.1yr.base
/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/test-20240812.v2.LR.F2010-FAMIP-hybrid.1yr.volc-cdf5

The overall speedup was 30.2 to 32.9 sypd or 8.9% faster.

[ndkeen]

Rob I don't think this issue is related to the one you pointed to

[rljacob]

Just "related" in the sense of another instance of bad i/o.