RFC: Event bus

[thewilkybarkid]

Problem

Libero needs to allow for disparate services to indirectly communicate with each other, so they can monitor and react to data changes.

Suggestion

• Use RabbitMQ (AMQP 0.9.1 standard + publisher confirms extension) to allow for services to reliably emit notifications that something has happened
• Define a standard for routing keys
  • Namespacing
• Services can act both in an upstream and downstream role
  • Upstream services own and send messages to their ‘topic’ exchange(s)
  • Downstream services bind their queue(s) to one or more exchanges
  • Downstream services consume messages from their queue(s)

Concerns

• RabbitMQ is an opinionated choice, but other existing options/standards don’t meet our requirements (AMQP has two flavours, and doesn’t a performant way of confirming publication)

[giorgiosironi]

Adding https://docs.google.com/document/d/1zTqwWQNOqrzVqmwnWUqKsHq8fHFLSByhTvMNrgBoIXI/edit here which briefly lists the open source alternatives to RabbitMQ that were evaluated (ZeroMQ and ActiveMQ), plus:

• requirements to be met
• the AMQP protocol and the RabbitMQ extensions necessary for fully reliable delivery
• the driver and libraries support for PHP and Python, the main programming languages we have been using so far during prototyping

[giorgiosironi]

Pointing out this analysis was originally scoped to Libero Publisher, but is not now potentially a Libero-wide decision.

[petereast]

We've been doing quite a bit of work looking into a generic event bus system for libero-reviewer. Part of this work has been in collaboration with PubSweet where we tried to agree with Hindawi on the interface for a generic event bus, so we could support the "event bus" pattern without needing to align on the specific technology (as Hindawi use AWS SQS and SNS -- not the choice for libero). Reviewer currently includes an implementation of this pattern using RabbitMQ, details of which can be found here: https://github.com/libero/reviewer/tree/master/lib/event-bus

[giorgiosironi]

Would be interesting to include here whatever decisions are made at the PubSweet level, as these are a coupling point. For example if https://github.com/libero/reviewer/blob/master/lib/event-bus/src/event-bus/index.ts#L5-L17 goes into PubSweet and Libero products align themselves between each other, then Pubsweet dictates the format of messages for all Libero products.