Tidying of queued messages

[adamcooke]

There are occasions when the queued messages table can have data within it which cannot be processed.

1. A queued message is locked but not removed from the queue (usually occurs when there's a database issue during processing.)

2. The messages associated with a queued message is no longer available which means they cannot be displayed in the web UI.

---

To improve this situation, we need to add a system for tidying queued messages which have been locked for a period of time which suggests they will not be unlocked. When we find stale locks we can either DELETE the queued message (so it is not retried automatically) or UNLOCK the queued message (so it can be retried again).

We need to be careful not to UNLOCK messages which might have been sent to HTTP or SMTP endpoints. We don't want to send messages twice erroneously.

Initially, I think DELETE-ing these messages will be safest. We can move to a model where we try to keep track of which stage a message de-queueing has reach and only UNLOCK if it has not reached the senders.

Additionally, as part of this, we should ensure the web UI can render the message queue when the backend message has been deleted. This is rare and, even more rare with the above in place, but it's worth just making sure it's resilient to this.