Unit tests fail due to ES TransportErrors

[amarjandu]

Observed twice so far:
https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/15364#L3440
https://gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/-/jobs/37941#L3357

Snippet of failure from link above:

ERROR: test_zero_tallies (indexer.test_hca_indexer.TestHCAIndexer)
Since duplicate notifications are subtracted back out of tally counts,
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/indexer/test_hca_indexer.py", line 130, in setUp
    self.index_service.create_indices(self.catalog)
  File "/builds/ucsc/azul/src/azul/indexer/index_service.py", line 289, in create_indices
    index = es_client.indices.get(index=index_name)
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 192, in get
    return self.transport.perform_request(
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 269, in perform_request
    self._raise_error(response.status, raw_data)
  File "/build/.venv/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [515229544/491.3mb], which is larger than the limit of [510027366/486.3mb], real usage: [515229544/491.3mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=5581228/5.3mb]')
----------------------------------------------------------------------

Linked discussion: https://ucsc-gi.slack.com/archives/C705Y6G9Z/p1643318850134000

[theathorn]

@hannes-ucsc : "I think this is caused by the new parent circuit breaker that was introduced in ES 7. Some index requests are close to the limit in size and therefore fail non-deterministically depending on the available memory. Disable the memory dependent circuit breaker or increase the limit so that it never trips during unit tests"

[hannes-ucsc]

Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.

elastic/elasticsearch#31767

[amarjandu]

This occurred again see https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/15716

elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [526780488/502.3mb], which is larger than the limit of [510027366/486.3mb], real usage: [526780488/502.3mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=13991976/13.3mb]')