[FEATURE REQUEST] Store CEDS emission data in one file per year?

[JiaweiZhuang]

The newly-added CEDS data are 140 GB in total. This increases the size of the default HEMCO data directory from ~70 GB to ~210 GB.

The problem is that 65 years (1950~2014) of data are stored in a single file. But most users would probably just need recent years. Would it be more reasonable to use one file per year? Other biggish data such as QFED are also stored in a per-year basis. This saves downloading time and also reduces the size of the tutorial AMI on AWS from 200+ GB to ~80 GB.

Breaking the CEDS data into 65 files (years) can be done by 3 lines of Python:

ds = xr.open_mfdataset(MAINDIR+'*-em-anthro_CMIP_CEDS_195001-201412.nc')
for year in range(1950, 2015):
    ds.sel(time=str(year)).to_netcdf('~/output/CEDS_{}.nc'.format(year))

See this notebook for a walk through.

[JiaweiZhuang]

PS: Found 2 typos in the HEMCO data wiki:

• Size of C2H6_2010/v2017-05 should be 540K, not 140 GB
• $ROOT/NEI201ek/2018-04 -> $ROOT/NEI2011ek/2018-04

Also, newly added CHEM_INPUTS/MODIS_LAI_201707/ is not uploaded to FTP.

[msulprizio]

Hi Jiawei. We can certainly split the CEDS files into yearly files to simplify data downloads. I will look into adding this change to 12.1.0. We are including other CEDS fixes in that version.

[JiaweiZhuang]

Thanks!

[msulprizio]

Also, the typos have been fixed on the HEMCO data wiki page.

I can access the MODIS LAI data at
ftp://ftp.as.harvard.edu/gcgrid/data/ExtData/CHEM_INPUTS/MODIS_LAI_201707/For_Olson_2001/. Can you not access that data?

[JiaweiZhuang]

Just found it. It turns out that I was searching inside ftp://ftp.as.harvard.edu/gcgrid/data/ExtData/GEOS_NATIVE/, which contains MODIS_LAI_200911 and MODIS_LAI_201204 but no MODIS_LAI_201707.

[msulprizio]

OK great!

[msulprizio]

A fix for this issue has now been pushed to the dev/12.1.0 branch in geos-chem-unittest. Yearly files can now be found in HEMCO/CEDS/v2018-08. In addition, the files in HEMCO/CEDS/v2018-04 have been compressed using nccopy -d1, bringing the directory size from 140 GB to 26 GB.