Satellite Processors designed for simple CMAQ comparisons.
- convert L2 or L3 satellite products to L3 on CMAQ grids
- 2-d species like total or tropospheric columns
- n-d vairables like averaging kernels and scattering weights.
- convert CMAQ concentrations to L3-like products
- Apply satellite averaging kernels to CMAQ concentrations to make satellite-like CMAQ
- Apply CMAQ to create alternative air mass factors to make CMAQ-like satellite products.
cmaqsatproc has an easy full suite approach
- Operates on local files or dynamically finds remote files
- User can specify input files from their disk.
- Queries NASA's Common Metadata Repository (CMR) or NOAA AWS
- Allows for spatial subsetting based on a simple box.
- User can specify the box based on lat/lon
- The CMAQ grid can be used to automatically define the box.
- Provides L2 access as a dataframe or makes Level 3 data as a dataset
- Simple instructions are provided to configure Google Colab.
This example requires OpenDAP access configured on the machine, but does not require any local files (satellite or CMAQ).
import cmaqsatproc as csp
GDNAM = '12US1'
date='2019-07-24'
readername = 'TropOMINO2' # or TropOMIHCHO, VIIRS_AERDB, ...
outpath = f'{readername}_{date}_{GDNAM}.nc'
cg = csp.open_griddesc(GDNAM)
satreader = csp.reader_dict[readername]
l3 = satreader.cmr_to_level3(
temporal=f'{date}T00:00:00Z/{date}T23:59:59Z',
bbox=cg.csp.bbox(), grid=cg.csp.geodf, verbose=9
)
l3.to_netcdf(outpath)
This example assumes you have downloaded satellite files. The code is largely
the same as the previous. Instead of cmr_to_level3, it the method uses glob
to make a list of files that it passes to paths_to_level3.
from glob import glob
import cmaqsatproc as csp
GDNAM = '12US1'
date='2019-07-24'
readername = 'VIIRS_AERDB' # or TropOMIHCHO, VIIRS_AERDB, ...
outpath = f'{readername}_{date}_{GDNAM}.nc'
cg = csp.open_griddesc(GDNAM)
satreader = csp.reader_dict[readername]
paths = sorted(glob('AERDB_L2_VIIRS_SNPP*.nc'))
l3 = satreader.paths_to_level3(
paths, bbox=cg.csp.bbox(), grid=cg.csp.geodf, verbose=9
)
l3.to_netcdf(outpath)
This example requires an output from one of the previous exmaples. The data from the level3 satellite output is combined with CMAQ to make a comparison.
import cmaqsatproc as csp
import xarray as xr
GDNAM = '12US1'
date='2019-07-24'
readername = 'TropOMINO2'
satreader = csp.reader_dict[readername]
l3 = xr.open_dataset(f'{readername}_{date}_{GDNAM}.nc')
qf = csp.open_ioapi(f'CCTM_CONC_{date}_{GDNAM}.nc')[['NO2']]
mf = csp.open_ioapi(f'METCRO3D_{date}_{GDNAM}.nc')
qf['DENS'] = mf['DENS']
qf['ZF'] = mf['ZF']
qf['PRES'] = mf['PRES']
# Create satellite according to CMAQ, and CMAQ according to satellite
overf = satreader.cmaq_process(qf, l3)
overf.to_netcdf(f'{readername}_{date}_{GDNAM}_CMAQ.nc')
- Spatial matching is pretty good
- For satellite products with pixel corners, fractional area weighting is used by default. Other options are avilable.
- Satellite AveragingKernels are averaged
- within a single day or overpass depending on configuration.
- within grid cells.
- CMAQ stratosphere is using one of several methods
- Removed according to the satellite averaging kernel.
- numpy
- xarray
- netcdf4
- pyproj
- pandas
- geopandas
- h5netcdf is required for s3 support
OpenDAP is supported through standard NetCDF-C support. If authentication is
required, configure .netrc and .dodsrc. The configuration is described
several places. Although urls tend to update, the links below are currently useful:
- https://disc.gsfc.nasa.gov/data-access
- https://docs.unidata.ucar.edu/netcdf-c/current/auth.html
- https://docs.opendap.org/index.php/DAP_Clients_-_Authentication
To summarize those resources, make a user-access-only .netrc file. Then, make
a .dodsrc file that points to the .netrc file and a .urs_cookies file.
The commands below achieve this goal, but will overwrite anything you already
have there:
touch ~/.netrc
touch ~/.urs_cookies
chmod 0600 ~/.netrc
cat << EOF > ~/.dodsrc
HTTP.NETRC=${HOME}/.netrc
HTTP.COOKIEJAR=${HOME}/.urs_cookies
EOF
# where <uid> and <password> are your Earthdata credentials
cat << EOF >> ~/.netrc
machine urs.earthdata.nasa.gov
login <uid>
password <password>
EOFNotes:
- I have only been able to make this work if the files are in the user home directory.
- netcdf4-python version 1.6 has trouble with opendap.
flowchart TB;
subgraph userinputs [User Options];
direction TB;
query_opts(query options);
localpaths[(local paths\nor urls)];
grid(CMAQ grid);
end
subgraph cmaqsatproc [ ];
direction TB;
csplabel[cmaqsatproc];
cmr_links[[cmr_links]];
xarray -->to_dataframe[[to_dataframe]]
xarray[[open_dataset]] -->to_level3[[to_level3]]
end
subgraph outputs
direction LR;
l3;
csv;
end
subgraph CMR [. Common Metadata Repository .];
direction TB;
CMR_API[[CMR API]];
NASA_DB[(NASA Database)];
NASA_DB <--> CMR_API;
end
grid -->to_level3
query_opts -->cmr_links;
cmr_links <-->CMR_API;
localpaths --> xarray;
cmr_links -- OpenDAP links--> xarray;
to_level3 --> l3[\Level3 NetCDF or CSV/];
to_dataframe --> csv[\Level2 CSV/];
style query_opts fill:#ff0,stroke:#333,stroke-width:4px
style userinputs fill:#fff,stroke:#000,stroke-width:1px
style outputs fill:#fff,stroke:#000,stroke-width:1px
style grid fill:#ff0,stroke:#333,stroke-width:4px
style localpaths fill:#ff0,stroke:#333,stroke-width:4px
style CMR fill:#ccc,stroke:#333,stroke-width:4px
style cmaqsatproc fill:#cefad0,stroke:#cefad0,stroke-width:4px
style csplabel fill:#cefad0,stroke:#cefad0,stroke-width:4px
Henderson, B. H. (2022, October 17). cmaqsatproc v2: Satellite data processing for CMAQ [Poster]. 21st Annual Community Modeling and Analysis System Conference, Chapel Hill, NC. https://www.cmascenter.org/conference/2022/agenda.cfm; direct link