How can I create a PDS3-like index table for PDS4 metadata?

[jordanpadams]

Use Case 1
The use case is always that one needs to divide up data sets by their metadata. While I don't know its original intent, many of current index files today contain many columns of useful metadata related to each observation (times, angles of illumination, lat/lon etc.), so even when I have downloaded all the data that I need already, it is just faster to divide up datasets based on an index file that already has all metadata of interest compared to looping over all labels that are next to the data and decide by that if for the current analysis I need that data.

Use Case 2
Another use case is simply to study the metadata itself for the whole datasets, to see if the parameter space for an idea one has is sufficiently covered in the dataset. It is convenient to have these files on-disk for being able to work with them offline. planetarypy already stores all PDS3 index file it supports in compressed parquet files so reading speed or file sizes are not an issue.

[tloubrieu-jpl]

@jordanpadams @alexdunnjpl , We could be able to do that with the summary/properties of a request /products/{bundle lidvid}/members/members. I am pretty sure that will not work now because that would return only the properties of the first page of results.

Do you think that is what is expected ? I am not familiar with PDS3 index.

[tloubrieu-jpl]

In addition, the request in production, for example https://pds.nasa.gov/api/search/1/products/urn:nasa:pds:msledl::1.1/members/members is failing because of a known bug which is fixed in the next release.

On my. test server:
http://localhost:8081/products/urn:nasa:pds:insight_rad::2.1/members/members

Returns:

But the list of properties is limited to those found in the first page of results (here 100 which is all of the result for this test case where 14 products are returned). If we would like that to work at a largest scale, we need to upgrade the service. The ticket NASA-PDS/registry-api#277

[pds-ops]

We have a CSV response format you can query to get the output you are looking for.

You can get all available metadata like this:

$ curl --get 'https://pds.nasa.gov/api/search/1/classes/bundles/urn:nasa:pds:insight_rad::2.1/members/members' \
    --header 'Accept: application/csv'

Or you can narrow on specific subset of properties like this:

$ curl --get 'https://pds.nasa.gov/api/search/1/classes/bundles/urn:nasa:pds:insight_rad::2.1/members/members?fields=lidvid,pds:Identification_Area.pds:title'
    --header 'Accept: application/csv'

[pds-ops]

Note: You may want to ping the JSON response with limit=0 first in order to get the result summary back and determine the number of hits and paginate as needed. The limit defaults to 100.

$ curl --get https://pds.nasa.gov/api/search/1/classes/bundles/urn:nasa:pds:insight_rad::2.1/members/members?limit=0 \
    --header 'Accept: application/json'

{
   "summary" : {
      "hits" : 44,
      "limit" : 0,
      "properties" : [
         "_package_id",
         "alternate_ids",
         "description",
         "insight:Observation_Information.insight:mission_phase_name",
         "insight:Observation_Information.insight:product_type",
         "insight:Observation_Information.insight:release_number",
         "insight:Observation_Information.insight:software_name",
         "insight:Observation_Information.insight:software_version_id",
         "insight:Observation_Information.insight:spacecraft_clock_count_partition",
         "insight:Observation_Information.insight:spacecraft_clock_start_count",
         "insight:Observation_Information.insight:spacecraft_clock_stop_count",
         "insight:Observation_Information.insight:start_local_mean_solar_time",
         "insight:Observation_Information.insight:start_sol_number",
         "insight:Observation_Information.insight:start_solar_longitude",
         "insight:Observation_Information.insight:stop_local_mean_solar_time",
         "insight:Observation_Information.insight:stop_sol_number",
         "insight:Observation_Information.insight:stop_solar_longitude",
         "lid",
         "lidvid",
         "ops:Data_File_Info.ops:creation_date_time",
         "ops:Data_File_Info.ops:file_name",
         "ops:Data_File_Info.ops:file_ref",
         "ops:Data_File_Info.ops:file_size",
         "ops:Data_File_Info.ops:md5_checksum",
         "ops:Data_File_Info.ops:mime_type",
         "ops:Harvest_Info.ops:harvest_date_time",
         "ops:Harvest_Info.ops:node_name",
         "ops:Label_File_Info.ops:creation_date_time",
         "ops:Label_File_Info.ops:file_name",
         "ops:Label_File_Info.ops:file_ref",
         "ops:Label_File_Info.ops:file_size",
         "ops:Label_File_Info.ops:md5_checksum",
         "ops:Tracking_Meta.ops:archive_status",
         "pds:Citation_Information.pds:description",
         "pds:Citation_Information.pds:publication_year",
         "pds:Document.pds:description",
         "pds:Document.pds:document_name",
         "pds:Document.pds:publication_date",
         "pds:Document_Edition.pds:edition_name",
         "pds:Document_Edition.pds:files",
         "pds:Document_Edition.pds:language",
         "pds:Document_File.pds:document_standard_id",
         "pds:Document_File.pds:file_name",
         "pds:Field_Character.pds:data_type",
         "pds:Field_Character.pds:description",
         "pds:Field_Character.pds:field_length",
         "pds:Field_Character.pds:field_location",
         "pds:Field_Character.pds:field_number",
         "pds:Field_Character.pds:name",
         "pds:Field_Character.pds:unit",
         "pds:File.pds:creation_date_time",
         "pds:File.pds:file_name",
         "pds:Identification_Area.pds:information_model_version",
         "pds:Identification_Area.pds:logical_identifier",
         "pds:Identification_Area.pds:product_class",
         "pds:Identification_Area.pds:title",
         "pds:Identification_Area.pds:version_id",
         "pds:Internal_Reference.pds:lid_reference",
         "pds:Internal_Reference.pds:reference_type",
         "pds:Investigation_Area.pds:name",
         "pds:Investigation_Area.pds:type",
         "pds:Modification_Detail.pds:description",
         "pds:Modification_Detail.pds:modification_date",
         "pds:Modification_Detail.pds:version_id",
         "pds:Observing_System_Component.pds:description",
         "pds:Observing_System_Component.pds:name",
         "pds:Observing_System_Component.pds:type",
         "pds:Primary_Result_Summary.pds:processing_level",
         "pds:Primary_Result_Summary.pds:purpose",
         "pds:Record_Character.pds:fields",
         "pds:Record_Character.pds:groups",
         "pds:Record_Character.pds:record_length",
         "pds:Science_Facets.pds:discipline_name",
         "pds:Table_Character.pds:description",
         "pds:Table_Character.pds:name",
         "pds:Table_Character.pds:offset",
         "pds:Table_Character.pds:record_delimiter",
         "pds:Table_Character.pds:records",
         "pds:Target_Identification.pds:name",
         "pds:Target_Identification.pds:type",
         "pds:Time_Coordinates.pds:start_date_time",
         "pds:Time_Coordinates.pds:stop_date_time",
         "product_class",
         "ref_lid_document",
         "ref_lid_instrument",
         "ref_lid_instrument_host",
         "ref_lid_investigation",
         "ref_lid_target",
         "title",
         "vid"
      ],
      "q" : "",
      "sort" : [],
      "start" : 0,
      "took" : 148
   }
}