Error in ingestion server tests: index audio_temporary not found

[obulat]

Description

While debugging the CI failure in #904, I saw an error in the logs of the ingestion server test:

integration_ingestion_server_1  | 2023-03-29 04:04:18,316 INFO indexer.py:469 - Index `audio-temporary` was deleted - data refresh complete! :tada:
integration_ingestion_server_1  | 2023-03-29 04:04:18,320 INFO indexer.py:271 - Sending callback request
integration_ingestion_server_1  | 2023-03-29 04:04:18,326 INFO indexer.py:273 - Response: {"message": "OK"}
integration_ingestion_server_1  | 2023-03-29 04:04:18,328 WARNING base.py:288 - PUT http://integration_es:9200/audio-temporary/_settings [status:404 request:0.010s]
integration_ingestion_server_1  | Process Process-10:
integration_ingestion_server_1  | 2023-03-29 04:04:18,330 INFO tasks.py:226 - Task 2331c5050a8b4972b095e652e51c9348 completed.
integration_ingestion_server_1  | Traceback (most recent call last):
integration_ingestion_server_1  |   File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
integration_ingestion_server_1  |     self.run()
integration_ingestion_server_1  |   File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
integration_ingestion_server_1  |     self._target(*self._args, **self._kwargs)
integration_ingestion_server_1  |   File "/ingestion_server/ingestion_server/indexer.py", line 259, in refresh
integration_ingestion_server_1  |     self.es.indices.put_settings(
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
integration_ingestion_server_1  |     return func(*args, params=params, headers=headers, **kwargs)
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/client/indices.py", line 889, in put_settings
integration_ingestion_server_1  |     return self.transport.perform_request(
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 466, in perform_request
integration_ingestion_server_1  |     raise e
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 427, in perform_request
integration_ingestion_server_1  |     status, headers_response, data = connection.perform_request(
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/connection/http_requests.py", line 216, in perform_request
integration_ingestion_server_1  |     self._raise_error(response.status_code, raw_data)
integration_ingestion_server_1  |   File "/venv/lib/python3.10/site-packages/elasticsearch/connection/base.py", line 328, in _raise_error
integration_ingestion_server_1  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
integration_ingestion_server_1  | elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [audio-temporary]', audio-temporary, index_or_alias)
integration_ingestion_server_1  | [2023-03-29 04:04:18 +0000] [7] [DEBUG] POST /task

This error is repeated a couple of times. You can see it in the logs for passing CI, too. I suspect that moving to a context manager for multiprocessing during ingestion without calling pool.join() at the end to wait for all tasks to end was failing because of this error.

Reproduction

1. Open the successful run of the CI/CD: https://github.com/WordPress/openverse/actions/runs/4550084588/jobs/8022800726.
2. Open the logs for "Print ingestion test logs" step.
3. See error.

Additional context

I set the priority to low because the tests still pass, but please raise the priority if this is actually causing more problems.

[sarayourfriend]

without calling pool.join() at the end to wait for all tasks to end was failing because of this error.

Fixed in cf9febd. Confirming that I cannot see the same error in recent runs of the ingestion server tests in CI.