Issue #13299 Asyncronous download of specific parameters to CSV and N… #65
Conversation
…etCSF is inconsistent - Insure lat/lon, if available, are included in JSON format - Remove stream name from external parameters in CSV and JSON formats
util/csvresponse.py
Outdated
| @@ -137,6 +137,13 @@ def _create_csv(self, dataset, filehandle): | |||
| # filter the dataset | |||
| dataset = dataset.drop(drop.intersection(dataset)) | |||
|
|
|||
| # rename the parameter without the stream_name prefix (12544 AC1) | |||
There was a problem hiding this comment.
for csv's and json they don't have the extra metadata for each variables that netcdf has to identifies the actual stream from which the parameter came from. netcdf can get away with removing the stream name from the variable because it has additional metadata attributes that help identify the true source of the data.
removing the stream name introduces a change in the interface of the data which may break user developed m2m tools. before implementing this change for csv's and json we probably should get feedback from the user community if this is what they want.
There was a problem hiding this comment.
Agreed. Will pull the name changes and update the ticket to request input/direction from the user community.
There was a problem hiding this comment.
I think this will also correct the test failure.
…etCSF is inconsistent - Insure lat/lon, if available, are included in JSON format - remove name change for externals in JSON and CSV packaging
| @@ -130,14 +130,18 @@ def _deployment_particles(self, ds, stream_key, parameters, external_includes): | |||
| if lat_data is not None and lon_data is not None: | |||
| data['lat'] = lat_data | |||
| data['lon'] = lon_data | |||
| params.extend(('lat', 'lon')) | |||
| # always include lat/lon | |||
| params.extend(('lat', 'lon')) | |||
There was a problem hiding this comment.
Is it correct to include lat/lon when we do not have values for them?
There was a problem hiding this comment.
lines 127 - 132 apply to gliders only and essentially rename GPS based lat/lon as simply lat or lon. Some non-glider data does have lat/lon, e.g. GI01SUMO_RID16_02_FLORTD000, which were not being reported. At this point, we are actually saying we want to include lat/lon; if they are not available, they are reported as missing in lines 161 - 163 and removed from the requested param list in 165.
I think it is more efficient to do this than to attempt to determine ahead of time if lat/lon are available.
There was a problem hiding this comment.
In netcdf_generator.py, external parameters associated with directly requested parameters are added to the list of parameters to be included with the request (i.e. not filtered out - see "params_to_include.append(parameter.name)"). Then they are renamed if applicable. In csvresponse.py, you are simply renaming the streams after the filtering is already done.
In other words, you seem to perform the equivalent of the NetCDF code for ticket 12544, but leave out the equivalent of the NetCDF code for ticket 12886. I don't know whether or not 12886 should apply to CSV.
…etCSF is inconsistent
Will update release notes once the code changes are approved.