Notebooks no longer work with V1.0 release

[jordanpadams]

🐛 Describe the bug

• Data in the registry is incorrect
• Queries go against pds-gamma instead of ops registry

[tloubrieu-jpl]

Blocked by:
NASA-PDS/registry-api#171
NASA-PDS/pds-api#173
NASA-PDS/registry-api#144
NASA-PDS/registry-api#152
NASA-PDS/registry-mgr#53

[jordanpadams]

@tloubrieu-jpl do we actually plan on fixing this during this sprint, of should we add this to the top of the release backlog?

[jordanpadams]

@jjacob7734 see PRs for some attempts at fixing this

[jordanpadams]

note: this should work for the latest registry / API on main (and maybe tagged for release?). not necessarily what is online (which doesn't work)

[jordanpadams]

feel free to ping @jimmie for some help here as well since he is digging into the tagged releases vs what is online vs what is going wrong

[jjacob7734]

I got the api_client.ipynb to work with a couple modifications (not yet checked in).

1. Changed configuration.host from:

https://pds-gamma.jpl.nasa.gov/api/

to:

https://pds.nasa.gov/api/search-en-gamma/1.1/

Reference: Based on information from the Slack discussion between @eddiesarevalo and @jimmie here: https://jpl.slack.com/archives/GCGR1R3A4/p1673301753096449

2. Changed:

api_response = api_instance.get_collection(...)

to the following:

from pds.api_client.api.collections_api import CollectionsApi
collections = CollectionsApi(api_client)
...
api_response = collections.get_collection(...)

Reference: API documentation at https://github.com/NASA-PDS/pds-api-client/blob/main/docs/source/quickstart/index.rst

[jjacob7734]

I got the api_client.ipynb to work with a couple modifications (not yet checked in).

1. Changed configuration.host from:

https://pds-gamma.jpl.nasa.gov/api/

to:

https://pds.nasa.gov/api/search-en-gamma/1.1/

Reference: Based on information from the Slack discussion between @eddiesarevalo and @jimmie here: https://jpl.slack.com/archives/GCGR1R3A4/p1673301753096449

2. Changed:

api_response = api_instance.get_collection(...)

to the following:

from pds.api_client.api.collections_api import CollectionsApi
collections = CollectionsApi(api_client)
...
api_response = collections.get_collection(...)

Reference: API documentation at https://github.com/NASA-PDS/pds-api-client/blob/main/docs/source/quickstart/index.rst

[jordanpadams]

@jjacob7734 awesome! can you create a PR for this?

[jordanpadams]

also @jjacob7734 even if we have PR, we may want to hold off until @jimmie deploys the latest registry to ops. the URL for the API should really be: https://pds.nasa.gov/api/search/1/

[jordanpadams]

looking at OVIRS notebooks, which is not working right now.

[jordanpadams]

looking at OVIRS notebooks, which is not working right now.

[jimmie]

familiarizing w/ API, specifically the new python version that Thomas made available. Seeing an issue where a collection by lidvid is returning a 404 but works w/ curl and otherwise works w/out version.

[jjacob7734]

It turns out the curl command was "working" because it was malformed and the registry was silently ignoring an invalid parameter, so not filtering the results by the LIDVID match (@alexdunnjpl opened a new ticket on this: NASA-PDS/registry-api#237). The root of the problem I was having is that the LIDVID I was trying to access does not exist in the production PDS registry. It exists in a particular profile of the registry that can be deployed locally for test. The new api_client does seem to be working fine if I give it a valid LIDVID.

[tloubrieu-jpl]

What to do with the notebook needs to be refined.

[tloubrieu-jpl]

Some part of the notebook is blocked by api issues.

We are not sure if the same data is still available in the registry.

[tloubrieu-jpl]

The short term solution is to use only the /products end-point.

[jordanpadams]

Status: working to change to just use the product API

[tloubrieu-jpl]

Keep the following notebooks:

• part1/...
• part2/ standalone

We need a clear description/organization of what they do for the users (in introduction of the notebook).

Remove part2/find_data

We move all the others in a draft. folder.

[jjacob7734]

@jordanpadams @tloubrieu-jpl recorded talk on PDS API: https://www.youtube.com/watch?v=iuPzeZMbhrs

[tloubrieu-jpl]

@jjacob7734 start to work on part1 notebook

[tloubrieu-jpl]

The expected outcome is to fully review the notebooks that we want and ideintify the blocker on the server side.
Then a ticket will be created for react to each of the removed blockers.

[tloubrieu-jpl]

@jjacob7734 is missing a way to apply the criteria, we will discuss that at the breakout.

[tloubrieu-jpl]

@jjacob7734 is testing the pagination in he notebook. He is going to test the performances.

We also need to deploy the update in the opensearch configuration to go beyond the 10000 limit.

[tloubrieu-jpl]

Blocked by:

• The updates are in branch search-api-notebook-6 and will be tested on the new registry deployed in production, version 1.2.0.
• The OpenSearch limit of 10000 need to be updated, Update staging and production Registry APIs to increase window limit registry-api#291
• Missing the properties end-points Product summary object has an incomplete "properties" set registry-api#277
• The SBN-PDS node needs to reload their data as well. See SBN PSI data sets ingested with past version of tools and fields are not searchable registry-api#288