[JOSS]: KeyError while retreiving ERA5 data using get_modeldata

[ashwinvis]

I got as far as this step, https://vergauwenthomas.github.io/MetObs_toolkit/examples/gee_example.html#Extracting-ERA5-timeseries

Input:

#Get the ERA5 data for a single station (to reduce data transfer)
your_station = your_dataset.get_station('vlinder02')

#Extract time series at the location of the station
ERA5_data = your_station.get_modeldata(modelname='ERA5_hourly',
                                      obstype='temp',
                                      startdt=None, #if None, the start of the observations is used
                                      enddt=None, #if None, the end of the observations is used
                                      )

Click below to view:

Traceback

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/test-metobs/lib/python3.11/site-packages/pandas/core/indexes/base.py:3802, in Index.get_loc(self, key, method, tolerance)
   3801 try:
-> 3802     return self._engine.get_loc(casted_key)
   3803 except KeyError as err:

File ~/test-metobs/lib/python3.11/site-packages/pandas/_libs/index.pyx:138, in pandas._libs.index.IndexEngine.get_loc()

File ~/test-metobs/lib/python3.11/site-packages/pandas/_libs/index.pyx:165, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5745, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5753, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'name'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[11], line 5
      2 your_station = your_dataset.get_station('vlinder02')
      4 #Extract time series at the location of the station
----> 5 ERA5_data = your_station.get_modeldata(modelname='ERA5_hourly',
      6                                       obstype='temp',
      7                                       startdt=None, #if None, the start of the observations is used
      8                                       enddt=None, #if None, the end of the observations is used
      9                                       )

File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/dataset.py:803, in Dataset.get_modeldata(self, modelname, modeldata, obstype, stations, startdt, enddt)
    801 # fill modell with data
    802 if modelname == "ERA5_hourly":
--> 803     Modl.get_ERA5_data(metadf=metadf,
    804                        startdt_utc=startdt_utc,
    805                        enddt_utc=enddt_utc,
    806                        obstype=obstype)
    808 else:
    809     Modl.get_gee_dataset_data(mapname=modelname,
    810                               metadf=metadf,
    811                               startdt_utc=startdt_utc,
    812                               enddt_utc=enddt_utc,
    813                               obstype=obstype)

File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/modeldata.py:464, in Modeldata.get_ERA5_data(self, metadf, startdt_utc, enddt_utc, obstype)
    429 def get_ERA5_data(self, metadf, startdt_utc, enddt_utc, obstype='temp'):
    430     """Extract timeseries of the ERA5_hourly dataset.
    431 
    432     The units are converted to the toolkit standard units.
   (...)
    462 
    463     """
--> 464     self.get_gee_dataset_data(mapname='ERA5_hourly',
    465                               metadf=metadf,
    466                               startdt_utc=startdt_utc,
    467                               enddt_utc=enddt_utc,
    468                               obstype=obstype)

File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/modeldata.py:400, in Modeldata.get_gee_dataset_data(self, mapname, metadf, startdt_utc, enddt_utc, obstype, target_unit_name, conv_expr)
    397 connect_to_gee()
    399 # Get data using GEE
--> 400 df = gee_extract_timeseries(metadf=metadf,
    401                             mapinfo=geeinfo,
    402                             startdt=startdt_utc,
    403                             enddt=enddt_utc,
    404                             obstype=obstype,
    405                             latcolname="lat",
    406                             loncolname="lon",
    407                             )
    409 if not df.empty:
    410     self._df_units[obstype] = geeinfo['band_of_use'][obstype]['units']

File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py:472, in gee_extract_timeseries(metadf, mapinfo, startdt, enddt, obstype, latcolname, loncolname)
    467     use_drive = True
    468 # =============================================================================
    469 # df to featurecollection
    470 # =============================================================================
--> 472 ee_fc = _df_to_features_point_collection(metadf)
    474 # =============================================================================
    475 # extract raster values
    476 # =============================================================================
    478 def rasterExtraction(image):

File ~/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py:194, in _df_to_features_point_collection(df)
    192 poi_geometry = ee.Geometry.Point([row["lon"], row["lat"]])
    193 #     construct the attributes (properties) for each point
--> 194 poi_properties = poi_properties = {"name": row["name"]}
    195 #     construct feature combining geometry and properties
    196 poi_feature = ee.Feature(poi_geometry, poi_properties)

File ~/test-metobs/lib/python3.11/site-packages/pandas/core/series.py:981, in Series.__getitem__(self, key)
    978     return self._values[key]
    980 elif key_is_scalar:
--> 981     return self._get_value(key)
    983 if is_hashable(key):
    984     # Otherwise index.get_value will raise InvalidIndexError
    985     try:
    986         # For labels that don't resolve as scalars like tuples and frozensets

File ~/test-metobs/lib/python3.11/site-packages/pandas/core/series.py:1089, in Series._get_value(self, label, takeable)
   1086     return self._values[label]
   1088 # Similar to Index.get_value, but we do not fall back to positional
-> 1089 loc = self.index.get_loc(label)
   1090 return self.index._get_values_for_loc(self, loc, label)

File ~/test-metobs/lib/python3.11/site-packages/pandas/core/indexes/base.py:3804, in Index.get_loc(self, key, method, tolerance)
   3802     return self._engine.get_loc(casted_key)
   3803 except KeyError as err:
-> 3804     raise KeyError(key) from err
   3805 except TypeError:
   3806     # If we have a listlike key, _check_indexing_error will raise
   3807     #  InvalidIndexError. Otherwise we fall through and re-raise
   3808     #  the TypeError.
   3809     self._check_indexing_error(key)

KeyError: 'name'

Debug session:

ipdb>  u
> /home/avmo/test-metobs/lib/python3.11/site-packages/metobs_toolkit/landcover_functions.py(194)_df_to_features_point_collection()
    192         poi_geometry = ee.Geometry.Point([row["lon"], row["lat"]])
    193         #     construct the attributes (properties) for each point
--> 194         poi_properties = poi_properties = {"name": row["name"]}
    195         #     construct feature combining geometry and properties
    196         poi_feature = ee.Feature(poi_geometry, poi_properties)

ipdb>  print(row)
index                                        vlinder02
network                                        Vlinder
lat                                          51.022379
lon                                           3.709695
call_name                                       Sterre
location                                          Gent
geometry                    POINT (3.709695 51.022379)
assumed_import_frequency               0 days 00:05:00
dataset_resolution                     0 days 00:05:00
lcz                                       Open midrise
altitude                                             7
water_100m                                         0.0
pervious_100m                                 0.428769
impervious_100m                               0.571231
water_250m                                         0.0
pervious_250m                                 0.535944
impervious_250m                               0.464056
Name: 0, dtype: object
ipdb>  q
1
#Get info

[vergauwenthomas]

@ashwinvis, I think this bug is already fixed in the master branch. I have tried but could not reproduce it with the master.

Could it be that your /home/avmo/test-metobs/lib/python3.11/site-packages/metobs_toolkit is not the (latest) master ?

(Since the version on PyPi is not synchronized with the github master, the PyPi version is lagging the master)

[vergauwenthomas]

Ref: openjournals/joss-reviews#5916

[ashwinvis]

You are right. This is fixed in the master branch. Closing.