Replay

Replay

Replay is a stage which replays the failed events on other stages. On each stage, when queue full or network congestion, the app will receive an advice event, and the adviced event will be written to kafka, the replay app will read the events from kafka and then replay it. The replay app will minimize event lose for the pipeline for occasional network or node failure.

The replay app have rate control capability to control the max rate send from the replay cluster. When the replay app send event failed, it will pause reading from kafka and put back the event into kafka, then wait a while and start replay again.

The replay app will decorate a tag js_rply (value=true) to indicate this event was sent from replay.

Inbound Replay

This is typically caused by some EPL changes, if a wrong but valid EPL is deployed at runtime, it will cause the Esper throws exception thus make the EsperProcessor itself paused, then it will trigger the advice event.

Outbound Replay

This most caused by downstream slow or total crashed, the event can not be pushed to the downstream, then the transport layer will send advice to Outbound messaging channel.

Deployment

The replay will connect to all stages of the pipeline, and it depends on kafka. Replay app itself poll messages from the kafka queue, itself did not have any inbound message channel. It just listen on 9999 port for monitoring.

Customization

The replay app can be extended by adding more replay topics. Each topic will include a:

1. A com.ebay.jetstream.event.channel.kafka.InboundKafkaChannel to read the events
2. A com.ebay.pulsar.replay.processor.ReplayInboundKafkaProcessor to process the events
3. An Outbound message channel to send events to other stages
4. A com.ebay.pulsar.replay.processor.ReplayAdviceProcessor to handle advice from the Outbound message channel
5. A com.ebay.jetstream.event.support.channel.RemoteController to pause the KafkaChannel when outbound message channel has advice

See Example