Intermittent KERIA Internal Server Errors (HTTP 500) when listing registries.

[lenkan]

I am getting intermittent internal server errors from KERIA when listing registries. It has happened the last couple of weeks. The stack trace is this:

keri-sandbox-keri-agent-1  | Sending TEL events to witnesses
keri-sandbox-keri-agent-1  | 2023-12-13 12:32:04 [FALCON] [ERROR] GET /identifiers/multisig/registries => Traceback (most recent call last):
keri-sandbox-keri-agent-1  |   File "falcon/app.py", line 365, in falcon.app.App.__call__
keri-sandbox-keri-agent-1  |   File "/keria/src/keria/app/credentialing.py", line 91, in on_get
keri-sandbox-keri-agent-1  |     state=asdict(registry.tever.state())
keri-sandbox-keri-agent-1  |   File "/keria/venv/lib/python3.10/site-packages/keri/vdr/credentialing.py", line 198, in tever
keri-sandbox-keri-agent-1  |     return self.reger.tevers[self.regk]
keri-sandbox-keri-agent-1  |   File "/keria/venv/lib/python3.10/site-packages/keri/vdr/viring.py", line 48, in __getitem__
keri-sandbox-keri-agent-1  |     raise ex  # reraise KeyError
keri-sandbox-keri-agent-1  |   File "/keria/venv/lib/python3.10/site-packages/keri/vdr/viring.py", line 43, in __getitem__
keri-sandbox-keri-agent-1  |     return super(rbdict, self).__getitem__(k)
keri-sandbox-keri-agent-1  | KeyError: 'ECKz_AJZmLmSHqwlztCFnH8Zi6rDBNFhxcJ_dq3FIs2T'
keri-sandbox-keri-agent-1  |
keri-sandbox-keri-agent-1  | Witness receipts complete, EEJPy1xYu3qwUdzVLwm0utcPYCFdJw4B_SBUC0Nq5TVU confirmed.

This happens when the registry does not have any backers. I do not have a clear reproduction yet.

[psteniusubi]

I'm seeing this too. Looks like this sometimes happens when invoking registries.list immediately after registry creation for a multisig identifier is completed.

My workaround is to retry once the registries.list operation after a small delay.

[lenkan]

I added a commented out step that triggers this in PR WebOfTrust/signify-ts#202 in signify.

[psteniusubi]

I have found a possible cause for this issue. Turns out there are a couple of long running operations that have metadata.depends set to an other long running operation: Registries.create() and Credentials.issue().

It looks like waiting for both operations to complete will make things more reliable and avoid the server side error.

How to resolve this issue? I think the server side of GET /operations/{aid} should also check dependencies. In the same way DELETE /operations/{aid} should delete dependencies. Possibly controlled by query parameter.

Below a sample operation from Registries.create() with metadata.depends set:

{
  "name": "registry.ELPrxeE2UtppYnCRh3JioNw3I2nY8a1FhnPle9prIov6",
  "metadata": {
    "anchor": {
      "i": "EA6OPxtGks_x-cY0HHD2w5dkJ4QWSb91KlK2s-ROnchQ",
      "s": "0",
      "d": "EA6OPxtGks_x-cY0HHD2w5dkJ4QWSb91KlK2s-ROnchQ"
    },
    "depends": {
      "name": "witness.ELPrxeE2UtppYnCRh3JioNw3I2nY8a1FhnPle9prIov6",
      "metadata": {
        "sn": 1
      },
      "done": false,
      "error": null,
      "response": null
    }
  },
  "done": false,
  "error": null,
  "response": null
}

[kentbull]

I believe I may have found the cause of this in KERIpy as stated in WebOfTrust/keripy#667.

The Tever.reload function expects a Serder instance yet is getting a RegStateRecord.

[pfeairheller]

Fixed with PR #197