Science museum script needs to implement get_should_continue to stop ingestion

[obulat]

Description

The ingestion for Science museum doesn't stop when the last page is reached, which causes an error due to response 400 for non-existent page.

Reproduction

1. Run an API request for page 50:

curl -X GET -H 'Accept: application/json' -v -i 'https://collection.sciencemuseumgroup.org.uk/search/?has_image=1&image_license=CC&page%5Bsize%5D=100&page%5Bnumber%5D=50&date%5Bfrom%5D=1750&date%5Bto%5D=1825'

2. Run a new API request, updating the URL's `page number` parameter to 51:

curl -X GET -H 'Accept: application/json' -v -i 'https://collection.sciencemuseumgroup.org.uk/search/?has_image=1&image_license=CC&page%5Bsize%5D=100&page%5Bnumber%5D=51&date%5Bfrom%5D=1750&date%5Bto%5D=1825'

3. See error: you will get a 400 response.

Solution

Each JSON response has a links dictionary with next property in it. We should stop the ingestion when next is null by overriding the get_should_continue function:

def get_should_continue(self, response_json):
   return response_json.get('links', {}).get('next') is not None

Resolution

• 🙋 I would be interested in resolving this bug.

[aqeelat]

@obulat On it.

[AetherUnbound]

We recently re-encountered this issue, so unfortunately it doesn't look like it's resolved 😞
https://airflow.openverse.engineering/log?execution_date=2022-11-01T00%3A00%3A00%2B00%3A00&task_id=ingest_data.pull_image_data&dag_id=science_museum_workflow&map_index=-1
We're looking into why this is happening now, since (preliminarily) it seems like it WordPress/openverse-catalog#868 should have addressed it.