Skip to content

Commit

Permalink
docs: move how to add new implementation doc from atlassian
Browse files Browse the repository at this point in the history
  • Loading branch information
navinkarkera committed Oct 16, 2023
1 parent 7dfa048 commit f91067d
Showing 1 changed file with 31 additions and 0 deletions.
31 changes: 31 additions & 0 deletions docs/how-tos/add-new-implementation.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
How to add a new concrete implementation of the event bus
=========================================================

Context
-------

Though the initial Open edX event bus was implemented using Kafka, the ultimate goal is for maintainers to be able to swap out different technologies like Redis or Pulsar. See `How to start using the Event bus <https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151>`_ for more information. Existing Concrete Implementations

- `Kafka <https://github.com/openedx/event-bus-kafka>`_
- `Redis Streams <https://github.com/openedx/event-bus-redis>`_

Producing
---------

There should be a producer class that inherits from `EventBusProducer <https://github.com/openedx/openedx-events/blob/cbb59f124ed84afacb9ec99baa82a86381370dcc/openedx_events/event_bus/__init__.py#L66>`_ in openedx-events.

The defined ``send`` method is meant to be called from within a signal receiver in the producing service.

Consuming
---------

At a high level, the consumer should be a process that takes the signals and events from the broker and emits the signal with the event. There should be a consumer class that inherits from `EventBusConsumer <https://github.com/openedx/openedx-events/blob/06635f3642cee4020d6787df68bba694bd1233fe/openedx_events/event_bus/__init__.py#L127>`_ in openedx-events.

The consumer class then needs to implement ``consume_indefinitely`` loop, which will stay running and listen to events as they come in.

If you are doing this within Django, one thing to be aware of is Django will not do any of the automatic connection cleanup that it usually does per request. This means that if your database restarts while the ``consume_indefinitely`` loop is running, Django may try to hold on to a defunct connection. To address this, we have included an `utility function <../../openedx_events/tooling.py#L323>`_ in openedx-events which needs to called before processing any signal. Checkout `consumer.py <https://github.com/openedx/event-bus-redis/blob/main/edx_event_bus_redis/internal/consumer.py>`_ in event bus redis implementation. We’re also resetting the RequestCache in the utility function, and there may be later, more comprehensive changes..

Abstraction tickets
-------------------

The known remaining work for a fully abstracted event bus is captured in the `Abstraction tickets <https://github.com/orgs/edx/projects/11/views/4?filterQuery=label%3Aevent-bus+-status%3ADone+abstraction>`_

0 comments on commit f91067d

Please sign in to comment.