fields parameter does not return values if more than one value is requested for CSV format

[tloubrieu-jpl]

🐛 Describe the bug

Request with fields parameter and response format CSV does not return multiple fields.

In addition, in our test example, the lidvid in the response are replicated

📜 To Reproduce

1. When I do the request:

% curl -G 'https://pds.nasa.gov/api/search/1.0/products' \
    --data-urlencode 'limit=1000' \
    --data-urlencode 'q=pds:Identification_Area.pds:product_class eq "Product_Observational"' \
    --data-urlencode 'fields=pds:Science_Facets.pds:discipline_name' \
    --data-urlencode 'fields=lidvid'  \
    --data-urlencode 'fields=ops:Harvest_Info.ops:node_name' 
    --header 'Accept:text/csv'

2. The result only contains the lidvid and empty fields for node_name and discipline:

"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044883359_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044883559_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044883759_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044883959_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044884159_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044884359_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044884559_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044884759_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044884959_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044885159_csv::1.0",,
"urn:nasa:pds:mess_xrs_derived:data_footprints:xrs_fp_2_044885359_csv::1.0",,

3. In addition the result is replicated, only different lidvid should show

🕵️ Expected behavior

All the three fields value should be returned.

The lidvid should not be replicated.

📚 Version of Software Used

version 1.0.1 in production.

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

---

🦄 Related requirements

⚙️ Engineering Details

[tloubrieu-jpl]

The lidvids are also replicated with the default response format, see request:

curl -G 'https://pds.nasa.gov/api/search/1.0/products' \
    --data-urlencode 'limit=1000' --data-urlencode 'q=pds:Identification_Area.pds:product_class eq "Product_Observational"' --data-urlencode 'fields=pds:Science_Facets.pds:discipline_name' --data-urlencode 'fields=lidvid'  --data-urlencode 'fields=ops:Harvest_Info.ops:node_name'  | json_pp

[jordanpadams]

@tloubrieu-jpl does this work if you put all the values in the same fields parameter (e.g. fields=field1,field2,field3? this seems like the more common approach for supporting multiple fields, versus fields=field1&fields=field2&fields=field3. it seems like trying to support multiple fields parameters in the same request seems counterintuitive from how I think about about URL query parameters, but maybe I am missing something?

[jordanpadams]

it almost seems like specifying multiple fields parameters would overwrite the values every time you reset it in the request.

[jordanpadams]

also, it looks like the documentation states to do fields=field1,field2,field3. in which case, I think this is actually a lower severity and actually a new feature request, not a bug.

[jordanpadams]

@al-niessner FYI ☝️

[al-niessner]

This behaves as expected:

$ curl --location --request GET 'http://localhost:8080/gid/bundle?fields=lid,lidvid' --header 'Accept: text/csv'
lid,lidvid
"urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1"

While this does not:

$ curl --location --request GET 'http://localhost:8080/gid/bundle' --data-urlencode 'fields=lid' --data-urlencode 'fields=lidvid' --header 'Accept: text/csv'
_package_id,description,lid,lidvid,ops:Data_File_Info.ops:creation_date_time,ops:Data_File_Info.ops:file_name,ops:Data_File_Info.ops:file_ref,ops:Data_File_Info.ops:file_size,ops:Data_File_Info.ops:md5_checksum,ops:Data_File_Info.ops:mime_type,ops:Harvest_Info.ops:harvest_date_time,ops:Harvest_Info.ops:node_name,ops:Label_File_Info.ops:creation_date_time,ops:Label_File_Info.ops:file_name,ops:Label_File_Info.ops:file_ref,ops:Label_File_Info.ops:file_size,ops:Label_File_Info.ops:md5_checksum,ops:Tracking_Meta.ops:archive_status,pds:Bundle.pds:bundle_type,pds:Bundle_Member_Entry.pds:lid_reference,pds:Bundle_Member_Entry.pds:member_status,pds:Bundle_Member_Entry.pds:reference_type,pds:Citation_Information.pds:author_list,pds:Citation_Information.pds:description,pds:Citation_Information.pds:doi,pds:Citation_Information.pds:publication_year,pds:File.pds:comment,pds:File.pds:file_name,pds:Identification_Area.pds:information_model_version,pds:Identification_Area.pds:logical_identifier,pds:Identification_Area.pds:product_class,pds:Identification_Area.pds:title,pds:Identification_Area.pds:version_id,pds:Internal_Reference.pds:lid_reference,pds:Internal_Reference.pds:reference_type,pds:Investigation_Area.pds:name,pds:Investigation_Area.pds:type,pds:Modification_Detail.pds:description,pds:Modification_Detail.pds:modification_date,pds:Modification_Detail.pds:version_id,pds:Observing_System.pds:name,pds:Observing_System_Component.pds:name,pds:Observing_System_Component.pds:type,pds:Primary_Result_Summary.pds:processing_level,pds:Primary_Result_Summary.pds:purpose,pds:Science_Facets.pds:discipline_name,pds:Science_Facets.pds:domain,pds:Stream_Text.pds:local_identifier,pds:Stream_Text.pds:name,pds:Stream_Text.pds:offset,pds:Stream_Text.pds:parsing_standard_id,pds:Stream_Text.pds:record_delimiter,pds:Target_Identification.pds:name,pds:Target_Identification.pds:type,pds:Time_Coordinates.pds:start_date_time,pds:Time_Coordinates.pds:stop_date_time,product_class,ref_lid_collection,ref_lid_collection_secondary,ref_lid_instrument,ref_lid_instrument_host,ref_lid_investigation,ref_lid_target,title,vid
"145d37d1-295f-41a3-9db6-a086ea90588f","[The InSight Radiometer data bundle consists of data in three collections: data_raw, data_calibrated, and data_derived. The bundle also includes the HP3/RAD Software Interface Specification in the HP3/RAD document collection., Pre-peer review version, First release, The collections urn:nasa:pds:insight_rad:data_calibrated and urn:nasa:pds:insight_rad:data_derived were added to this bundle with InSight Release 1b., Changed Observing_System_Component name in this label from RAD to RADIOMETER to match context product name. Expanded Citation_Information description.]","urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1","2020-01-03T17:58:09Z","readme.txt","/data/readme.txt","1114","192de32c12437c180a9e14d60fe4b89a","text/plain","2022-07-07T20:37:52.521363Z","PDS_ENG","2020-01-15T17:40:30Z","bundle_insight_rad.xml","/data/bundle_insight_rad.xml","6805","adfd86bbf2573c37d862e27e08f332db","archived","Archive","[urn:nasa:pds:insight_rad:data_raw, urn:nasa:pds:insight_rad:data_calibrated, urn:nasa:pds:insight_rad:data_derived, urn:nasa:pds:insight_documents:document_hp3rad]","[Primary, Primary, Primary, Secondary]","[bundle_has_data_collection, bundle_has_data_collection, bundle_has_data_collection, bundle_has_document_collection]","InSight RAD Science Team","The InSight Radiometer data bundle consists of data in three collections: data_raw, data_calibrated, and data_derived. The bundle also includes the HP3/RAD Software Interface Specification in the HP3/RAD document collection.","10.17189/1517568","2018-01-01T00:00:00Z","Introduction to the bundle","readme.txt","1.11.0.0","urn:nasa:pds:insight_rad","Product_Bundle","Mars InSight Lander Radiometer Data Archive","2.1","[urn:nasa:pds:context:investigation:mission.insight, urn:nasa:pds:context:instrument_host:spacecraft.insight, urn:nasa:pds:context:instrument:radiometer.insight, urn:nasa:pds:context:target:planet.mars]","[bundle_to_investigation, is_instrument_host, is_instrument, bundle_to_target]","InSight","Mission","[Pre-peer review version, First release, The collections urn:nasa:pds:insight_rad:data_calibrated and urn:nasa:pds:insight_rad:data_derived were added to this bundle with InSight Release 1b., Changed Observing_System_Component name in this label from RAD to RADIOMETER to match context product name. Expanded Citation_Information description.]","[2018-02-01T00:00:00Z, 2019-04-22T00:00:00Z, 2019-06-11T00:00:00Z, 2020-01-02T00:00:00Z]","[0.1, 1.0, 2.0, 2.1]","InSight","[InSight, RADIOMETER]","[Spacecraft, Instrument]","[Raw, Calibrated, Derived]","Science","Geosciences","Surface","BUNDLE.DESCRIPTION","Introduction to the Radiometer Data Bundle","0","7-Bit ASCII Text","Carriage-Return Line-Feed","Mars","Planet","2018-05-05T00:00:00Z","3000-01-01T00:00:00.000Z","Product_Bundle","[urn:nasa:pds:insight_rad:data_calibrated, urn:nasa:pds:insight_rad:data_derived, urn:nasa:pds:insight_rad:data_raw]","urn:nasa:pds:insight_documents:document_hp3rad","urn:nasa:pds:context:instrument:radiometer.insight","urn:nasa:pds:context:instrument_host:spacecraft.insight","urn:nasa:pds:context:investigation:mission.insight","urn:nasa:pds:context:target:planet.mars","Mars InSight Lander Radiometer Data Archive","2.1"

and if we look into the logs we find that curl did this:

2022-07-19 16:00:30.300 DEBUG 54 --- [nio-8080-exec-7] o.a.coyote.http11.Http11InputBuffer      : Received [GET /gid/bundle HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.68.0
Accept: text/csv
Content-Length: 24
Content-Type: application/x-www-form-urlencoded

fields=lid&fields=lidvid]

which is not equivalent to fields=lid,lidvid. As @jordanpadams already noted, the API documentation wants it as fields=field1,field2.

I tried this --data-urlencode 'fields=lid.lidvid' and it too fails. I looked at the logs and found this:

2022-07-19 16:06:06.821 DEBUG 54 --- [nio-8080-exec-8] o.a.coyote.http11.Http11InputBuffer      : Received [GET /gid/bundle HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.68.0
Accept: text/csv
Content-Length: 17
Content-Type: application/x-www-form-urlencoded

fields=lid.lidvid]

It should look like this for it to work:

2022-07-19 16:08:38.711 DEBUG 54 --- [nio-8080-exec-9] o.a.coyote.http11.Http11InputBuffer      : Received [GET /gid/bundle?fields=lid,lidvid HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.68.0
Accept: text/csv

]

This suggests that --data-urlencode in curl is not compatible with spring-boot-fox-swagger-something. Given how swagger.yml is configured this does make sense. We tell swagger to find all this stuff as part of the URL not as data sent along with URL. May be able to create a URL that does it but we do not currently support it.

Hope this helps.

[tloubrieu-jpl]

Hi @al-niessner ,

I looked a bit more into this issue and put aside the usage of --data-urlencode.

I am thinking the problem is on 'complex' fields such as ops:Data_File_Info.ops:creation_date_time. The code works fine for simple fields such as lid, title.

So far I am also seeing the issue only with response format text/csv.

See this request as example:

curl -X GET 'https://pds.nasa.gov/api/search/1.0/products?limit=1000&q=pds%3AIdentification_Area.pds%3Aproduct_class%20eq%20%22Product_Observational%22&fields=pds:Science_Facets.pds:discipline_name,lidvid,ops:Harvest_Info.ops:node_name' --header 'Accept:text/csv'

Do you remember there is a trick with the property name ops:Harvest_Info.ops:node_name in the APi being ops:Harvest_Info/ops:node_name in opensearch ? The/ in opensearch are replace with . in the API. Could that be the issue ?

Thanks,

[al-niessner]

@tloubrieu-jpl

Yes, I can easily replicate:

$ curl --location --request GET 'http://localhost:8080/gid/bundle?fields=lid,lidvid,ops:Data_File_Info.ops:creation_date_time' --header 'Accept: text/csv'
lid,lidvid,ops:Data_File_Info/ops:creation_date_time
"urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1",

[tloubrieu-jpl]

Good, thanks @al-niessner

[al-niessner]

@tloubrieu-jpl

Works now

$ curl --location --request GET 'http://localhost:8080/gid/bundle?fields=lid,lidvid,ops:Data_File_Info.ops:creation_date_time' --header 'Accept: text/csv'
lid,lidvid,ops:Data_File_Info.ops:creation_date_time
"urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1","2020-01-03T17:58:09Z"

[tloubrieu-jpl]

@gxtchen this should be tested from:

the latest stable version of the registry-api (1.0.2) which can be deployed using the latest stable registry docker compose (1.0.2)
the latest development version should also be tested but the CICD did not work on registry-api

[gxtchen]

@jimmie I am using registry-api-service 1.0.2, I am not getting the ops:Data_File_Info.ops:creation_date_time
% curl --location --request GET 'http://localhost:8080/products?fields=lid,lidvid,ops:Data_File_Info.ops:creation_date_time' --header 'Accept: text/csv'
lid,lidvid,ops:Data_File_Info/ops:creation_date_time
"urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1",

[al-niessner]

@jimmie I am using registry-api-service 1.0.2, I am not getting the ops:Data_File_Info.ops:creation_date_time % curl --location --request GET 'http://localhost:8080/products?fields=lid,lidvid,ops:Data_File_Info.ops:creation_date_time' --header 'Accept: text/csv' lid,lidvid,ops:Data_File_Info/ops:creation_date_time "urn:nasa:pds:insight_rad","urn:nasa:pds:insight_rad::2.1",

@gxtchen @jimmie

This has been addressed via #173 and will be deployed in version 1.0.3 as I understand it.