ERA5 CDS requests which return a mixture of ERA5 and ERA5T data

[lukas-rokka]

Description

As reported here, CDS will return an dataset with an extra expver dimension when a call spans over an date range that consist of both ERA5 and preliminary ERA5T data. This is not handled in Atlite.

expver 1 : ERA5 data, that has been quality checked.
expver 5 : ERA5T preliminary data (the three last months).

Expected Behavior

The expver dimension should be removed.

A bonus would be to report the date when the ERA5T (expver 5) data starts as this data might get updated/corrected later.

Fix

An easy fix is that has worked in all my use cases:
cutout.data = cutout.data.reduce(np.nansum, 'expver')

Guess it could be done somewhere earlier in the preparation of the data, i.e. before any derived variables are calculated. Having the extra expver dimension also doubles the memory footprint, so it can be good to be thoughtful about this also when/if splitting cds calls in the time dimension

[zoltanmaric]

I also ran into this issue today. The reduce(np.nansum, 'expver') solution is not completely robust - for example, if you have data spanning 2 consecutive years, then the values for the previous year will be non-NaN for both expver=1 and expver=5, meaning np.nansum will then double all values for the previous year.

What's worked for me (and is much faster than the reduce method) was the solution proposed here:

cutout.data = cutout.data.sel(expver=1).combine_first(cutout.data.sel(expver=5))

[zoltanmaric]

@fneum you seem to be the most active contributor on atlite. I could prepare a pull request that does something like master...zoltanmaric:atlite:patch-1 - would you be willing to review and merge if I make it nice? :)

[fneum]

You probably mean @FabianHofmann :) but sure, a PR is welcome!

[zoltanmaric]

Whoopsie, yeah, I did mean Fabian Hofmann :) Sorry about that. Alright, I'll prepare something.

[zoltanmaric]

Draft pull request: #261