this looks great to me, I would love to see and extra [ ] for testing performance, I guess we needs some sort of performance suite.

Note that there is some abstraction of streams coming in the redis-rb library. We don't need it for this use case, but could refactor based on it in the future.

Note that there is a bug in released Redis where listening for updates on streams which have been emptied can cause unpredictable results. At this point, I'm not sure how much it would impact message_bus, but it's worth keeping an eye on: redis/redis@c1e9186

The reason that we can't increase the duration for which we block listening for messages on streams is that we rely on regularly polling a separate key which stores our subscription state in order to detect the condition where Redis has been emptied, triggering us to wind back to read the stream from the start again.

There are two possible avenues to resolve this:

• Redis could inform us that a stream has been deleted such that we know to read from the start of it. I have filed Blocking XREAD on streams can't detect the stream being deleted redis/redis#5625 with this feature request. Even if implemented as requested, this has flaws:
  • A stream could be deleted and re-created in the interval between XREAD calls by a client holding a message pointer, who would then not be able to detect the requirement to roll back.
  • It's plausible that the integer ID keys be deleted while a Stream persists, which would cause publications to fail (streams are append-only).
  • It's likely that deletion of a stream would not trigger deletion of the subscription key, so we would not correctly wind back.
  • Others?
• We could include a vector clock in the stream message IDs such that our pointers will be valid even if streams get re-created, and we thus never have to explicitly wind back. This solution is more robust, because it doesn't come unstuck in the case where the stream dies and is repopulated in-between blocking XREAD calls, and it also removes the assumption that the stream and the subscription key are maintained or removed atomically. It does, however, require message_bus to abandon the idea of incrementing integer IDs in favour of more complex opaque IDs.

This is the biggest blocker to proceeding with this PR right now.

@benlangfeld do you feel we should keep this open? I am totally open to adding more backend implementations its just that we have been sitting on this for way too long.