CMORizer for NASA MERRA reanalysis #3039

axel-lauer · 2023-02-06T08:28:24Z

Description

This PR adds scripts for downloading and formatting selected variables from the NASA MERRA reanalysis (not to be confused with MERRA2). As not all variables of interest are contained in the CREATE-IP version of this dataset, the original files provided by NASA are used. These are provided in .hdf-eos format only.

This PR also fixes a small bug in the shared function format_time (esmvaltool/cmorizers/data/formatters/utilities.ncl) that prevented all steps of the time coordinate to be initialized correctly. This fix is needed for this formatting script to work properly.

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
~~- [ ] 🛠 Any changed dependencies have been added or removed correctly~~
🛠 All checks below this pull request were successful

New or updated data reformatting script

🛠 Documentation is available
🛠 The dataset has been added to the CMOR check recipe
🧪 Numbers and units of the data look physically meaningful

valeriupredoi

cheers muchly @axel-lauer 🍺 just a quick observation (pun intended) from me. Also - do we want to tell @glpotter we are also working on this? He is involved with our MERRA2 data handling and CREATE-IP project too 👍

valeriupredoi · 2023-02-16T13:06:38Z

esmvaltool/cmorizers/data/formatters/datasets/merra.ncl

+        datestr = datestr + tostring(mm)
+
+        fname = systemfunc("ls " + input_dir_path + "MERRA???.prod.assim." + \
+                           SOURCEFILE(vv) + datestr + ".hdf")


I see the file extension is hardwired here, but the downloaded accepts options for nc or nc4 as well, is there a file handler anywhere else that is converting/extracting those to HDF5 or MERRA will never come in as netCDF?

The original MERRA data are only available in .hdf format. Unfortunately... Since the stuff is archived, I do not expect this to change... But I am happy to change ".hdf" to ".*" if you like that better.

aha gotcha! But then I'd not turn on the download of any nc or nc4 files at download point (via cmd line option) since those files will be ignored, if they do get downloaded at all - either that, or add an "*" option here, but in that case will the file handling have to change inside the NCL code ie to load and process nc vs HDF5? (sorry, I still don't speak any NCL 😁 )

axel-lauer · 2023-02-16T13:14:26Z

cheers muchly @axel-lauer 🍺 just a quick observation (pun intended) from me. Also - do we want to tell @glpotter we are also working on this? He is involved with our MERRA2 data handling and CREATE-IP project too 👍

I am aware of this. Unfortunately, The CREATE-IP version of the MERRA data (at least the one that is published to ESGF) does not contain the cloud variables I am interested in...

remi-kazeroni

Thanks for your contribution, @axel-lauer! The code looks good to me from a technical point of view. I just have 2 minor comments you might want to check. I have run the downloader, cmorizer and recipe_check_obs on data for one year and all works fine.

Before that PR is merged, could you please move the raw and CMORized data to the right folder on Levante? I assume you have already downloaded the large dataset and CMORized all years available.

Edit: the remaining codacy issues can be ignored.

esmvaltool/recipes/examples/recipe_check_obs.yml

remi-kazeroni · 2023-08-16T14:30:35Z

esmvaltool/cmorizers/data/formatters/datasets/merra.ncl

+
+  ; Global attributes
+  SOURCE = "https://goldsmr3.gesdisc.eosdis.nasa.gov/data/MERRA_MONTHLY/"
+  REF = "doi: "


Would it be needed to add the doi here?

I think that is fine, because the actual doi of the variable written to the output file will added at the time the file is written (that is because every variable migh have a different doi). I can see that this is confusing and moved the definition of REF to where the output file is actually written.

Co-authored-by: Rémi Kazeroni <[email protected]>

axel-lauer · 2023-08-17T05:23:04Z

Before that PR is merged, could you please move the raw and CMORized data to the right folder on Levante? I assume you have already downloaded the large dataset and CMORized all years available.

Done.

hb326

@axel-lauer: looks good from my side! I checked some of the available files on Levante, and also ran the formatter to see if it would work. I only found one little thing that I commented on. Otherwise it is good to be merged in my opinion! :)

hb326 · 2023-08-18T12:08:59Z

esmvaltool/cmorizers/data/formatters/datasets/merra.ncl

+  base = getenv("cmor_tables")
+
+  CMOR_TABLE = getenv("cmor_tables") + \
+    (/"/cmip5/Tables/CMIP5_" + MIP, "/cmip5/Tables/CMIP5_" + MIP,  \  ; 3d asm


Here, I think the expression "getenv("cmor_tables")" could be replaced by "base" which is defined two lines earlier.

Thanks for checking! You are totally right, this is indeed redundant. I deleted the definition of base as this variable is not used anywhere.

@valeriupredoi

…_RTW * recipe_test_workflow_prototype: (237 commits) Add version to dataset in python example recipe to avoid "Unknown file format" issue on JASMIN (#3322) CMORizer for NASA MERRA reanalysis (#3039) Add `OBS-maintainers` team to documentation on OBS data maintenance and CMORizer reviews (#3335) Fixed provenance tracking for NCL multipanel PNGs (#3332) Cmorizer for NOAA-CIRES-20CR v3 reanalysis (clt, clwvi, hus, prw, rlut, rlutcs, rsut, rsutcs) (#3137) [Condalock] Update Linux condalock file (#3321) Slight refactoring of diag `galytska23/select_variables_for_tigramite.py` for generality and portability (for Changelog v2.10: authors: @valeriupredoi and @egalytska) (#3298) Removed recipe_carvalhais14nat from list of broken recipes (#3319) add Romain Beucher to CITATION as contributor (#3318) update `mamba` version in readthedocs configuration docs builds (#3310) [Github Actions] Compress all bash shell setters into one default option per workflow (#3315) [condalock] update conda lock creation Github Action workflow and ship updated (bot-generated) conda-lock file (#3307) Allow NCL unit conversion `kg s-1` -> `GtC y-1` (#3300) Add list of failing recipes for v2.9.0 release (#3294) Update diag_shapeselect.py to work with shapely v2 (#3283) Update release schedule after release of v2.9.0 (#3289) Add merge instructions to release instructions (#3292) Made sklearn test backwards-compatible with sklearn < 1.3 (#3285) Add release notes for v2.9 (#3266) Add release notes for v2.9 (#3266) ...

Co-authored-by: Rémi Kazeroni <[email protected]>

axel-lauer added 4 commits February 2, 2023 15:05

added first version of MERRA CMORizer

7f3d770

updated merra.ncl

a10de44

updated wget.py

5c0aab3

added attribute positive to radiation variables

50be9b3

axel-lauer added observations looking for technical reviewer looking for scientific reviewer Data labels Feb 6, 2023

axel-lauer requested review from schlunma and remi-kazeroni February 6, 2023 08:28

axel-lauer changed the title ~~CMORier for NASA MERRA reanalysis~~ CMORizer for NASA MERRA reanalysis Feb 6, 2023

Merge branch 'main' into cmorizer_nasa_merra_ncl

946d413

valeriupredoi reviewed Feb 16, 2023

View reviewed changes

remi-kazeroni added in technical review and removed looking for technical reviewer labels Aug 16, 2023

Merge branch 'main' into cmorizer_nasa_merra_ncl

d777746

remi-kazeroni added approved by technical reviewer and removed in technical review labels Aug 16, 2023

remi-kazeroni approved these changes Aug 16, 2023

View reviewed changes

axel-lauer and others added 2 commits August 17, 2023 07:15

made code easier to read

c2d5fb4

Update esmvaltool/recipes/examples/recipe_check_obs.yml

9e8e817

Co-authored-by: Rémi Kazeroni <[email protected]>

fixed typo in review suggestion

67cc895

remi-kazeroni closed this Aug 17, 2023

remi-kazeroni reopened this Aug 17, 2023

added variable clwvi to MERRA output

ff42457

hb326 added in scientific review and removed looking for scientific reviewer labels Aug 18, 2023

updated merra.ncl

6f0184f

added clwvi to docu and recipe_check_obs.yml

1316e29

hb326 requested changes Aug 18, 2023

View reviewed changes

Update merra.ncl

96ea358

hb326 added approved by scientific reviewer and removed in scientific review labels Aug 18, 2023

Merge branch 'main' into cmorizer_nasa_merra_ncl

c6e303e

remi-kazeroni requested a review from a team as a code owner August 18, 2023 13:31

hb326 approved these changes Aug 18, 2023

View reviewed changes

remi-kazeroni added this to the v2.10.0 milestone Aug 18, 2023

remi-kazeroni merged commit b244fef into main Aug 18, 2023
1 of 2 checks passed

remi-kazeroni deleted the cmorizer_nasa_merra_ncl branch August 18, 2023 14:03

remi-kazeroni mentioned this pull request Aug 14, 2023

Extension of available reanalysis data for ESMValTool #2851

Open

10 tasks

jvegreg pushed a commit that referenced this pull request Jan 14, 2024

CMORizer for NASA MERRA reanalysis (#3039)

ef89d3e

Co-authored-by: Rémi Kazeroni <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMORizer for NASA MERRA reanalysis #3039

CMORizer for NASA MERRA reanalysis #3039

axel-lauer commented Feb 6, 2023 •

edited by remi-kazeroni

Loading

valeriupredoi left a comment

valeriupredoi Feb 16, 2023 •

edited

Loading

axel-lauer Feb 16, 2023

valeriupredoi Feb 16, 2023

axel-lauer commented Feb 16, 2023

remi-kazeroni left a comment •

edited

Loading

remi-kazeroni Aug 16, 2023

axel-lauer Aug 17, 2023

axel-lauer commented Aug 17, 2023

hb326 left a comment

hb326 Aug 18, 2023

axel-lauer Aug 18, 2023

CMORizer for NASA MERRA reanalysis #3039

CMORizer for NASA MERRA reanalysis #3039

Conversation

axel-lauer commented Feb 6, 2023 • edited by remi-kazeroni Loading

Description

Checklist

New or updated data reformatting script

valeriupredoi left a comment

Choose a reason for hiding this comment

valeriupredoi Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

axel-lauer Feb 16, 2023

Choose a reason for hiding this comment

valeriupredoi Feb 16, 2023

Choose a reason for hiding this comment

axel-lauer commented Feb 16, 2023

remi-kazeroni left a comment • edited Loading

Choose a reason for hiding this comment

remi-kazeroni Aug 16, 2023

Choose a reason for hiding this comment

axel-lauer Aug 17, 2023

Choose a reason for hiding this comment

axel-lauer commented Aug 17, 2023

hb326 left a comment

Choose a reason for hiding this comment

hb326 Aug 18, 2023

Choose a reason for hiding this comment

axel-lauer Aug 18, 2023

Choose a reason for hiding this comment

axel-lauer commented Feb 6, 2023 •

edited by remi-kazeroni

Loading

valeriupredoi Feb 16, 2023 •

edited

Loading

remi-kazeroni left a comment •

edited

Loading