Registry-Sweepers Error: contained no hits when hits were expected

[sjoshi-jpl]

💡 Description

The following error has been occurring for ATM and GEO registry-sweeper tasks that has triggered multiple notifications during every Lambda run. Please take a look :

GEO-PROD

Error found in log group '/ecs/pds-geo-prod-registry-sweeper-task':

Timestamp (UTC): 2023-09-03 13:46:48.295000
Log Stream: ecs/pds-geo-prod-registry-sweeper-container/8f0a15e6c18d40b59214433c6abbe05f
Error Message: 2023-09-03 13:46:48,295::pds.registrysweepers.utils.db::ERROR::Response for query 346d70 contained no hits when hits were expected.  Returned data is incomplete.  Response was: {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZBSjMxUGlTSFNaS3c1WUd0b0pvZmtBAAAAAAAAPJgWekZYaWNTdFRRX3k3X3NFYWNSRGZlZxZibE9CX0F6SVRyR1R0RHlYSHBNb2F3AAAAAAAAOVEWdW5nTDhrYTJRSmVkdnQzWnAxaDNWZxZBSjMxUGlTSFNaS3c1WUd0b0pvZmtBAAAAAAAAPJcWekZYaWNTdFRRX3k3X3NFYWNSRGZlZw==', 'took': 2, 'timed_out': False, 'terminated_early': False, '_shards': {'total': 3, 'successful': 3, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 3460390, 'relation': 'eq'}, 'max_score': 1.0, 'hits': []}}

ATM-PROD

Error found in log group '/ecs/pds-atm-prod-registry-sweeper-task':

Timestamp (UTC): 2023-09-03 13:29:26.877000
Log Stream: ecs/pds-atm-prod-registry-sweeper-container/5ab74f8d4b4d45a6b19cda6cfb953956
Error Message: 2023-09-03 13:29:26,877::pds.registrysweepers.utils.db::ERROR::Response for query 945e60 contained no hits when hits were expected.  Returned data is incomplete.  Response was: {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZ3UktlUnNsdVEzU3g2eGx6SVpYaFhRAAAAAAAAACMWdUltaGl1SG9UTU9rSzNWYmo3QTRpdxZ3UktlUnNsdVEzU3g2eGx6SVpYaFhRAAAAAAAAACQWdUltaGl1SG9UTU9rSzNWYmo3QTRpdxZWc180MnowYVFwR2hhb25hd0VncFpRAAAAAAAAAAwWejFZSERmV0xSN2kzclBtNjE1c0c1QQ==', 'took': 1, 'timed_out': False, 'terminated_early': False, '_shards': {'total': 3, 'successful': 2, 'skipped': 0, 'failed': 1, 'failures': [{'shard': -1, 'index': None, 'reason': {'type': 'illegal_state_exception', 'reason': 'node [z1YHDfWLR7i3rPm615sG5A] is not available'}}]}, 'hits': {'total': {'value': 649002, 'relation': 'eq'}, 'max_score': 1.0, 'hits': []}}

[alexdunnjpl]

Self-note - check whether or not scroll_id is being updated on every request, as it should be

[alexdunnjpl]

ATM-PROD appears to be on the OpenSearch side - see failure

}
  'shard': -1,
  'index': None,
  'reason': {
    'type': 'illegal_state_exception',
    'reason': 'node [z1YHDfWLR7i3rPm615sG5A] is not available'
  }
}

@tloubrieu-jpl @sjoshi-jpl I'm not sure where to take this one, as it appears to be a failure of one of the shards associated with that node. Any ideas?

GEO-PROD is a bit weirder - it's not showing any shard failures, nor any errors. I'll introduce some improved logging, hopefully that yields new useful information.

[jordanpadams]

@sjoshi-jpl to look at this closer to see if the shard failure is happening regularly.

[alexdunnjpl]

Logging improvements introduced in #72 , #73
572ac41
81cb14a

Timeout thresholds have also been extended, though it won't be clear for a little while whether or not they're relevant to the timeouts in question.

[alexdunnjpl]

@sjoshi-jpl @jordanpadams I'm fairly certain this is resolved by #77

Closing on that basis. @sjoshi-jpl would you please remove any exclusion rules in the error log escalation lambda? We can re-open if this issue reappears.

[sjoshi-jpl]

@alexdunnjpl done. This issue statement has been removed from the sweepers lambda exceptions list and if they do occur again we should know.