Switch to pubsub based event streams from event scanning from logs using geth

[rnbguy]

Surfaced from @informalsystems audit of Althea Gravity Bridge at commit 19a4cfe

severity: Informational
type: Restructuring proposal
difficulty: Medium

Involved artifacts

• Orchestrator

Description

These functions here poll on Ethereum events

• orchestrator/orchestrator/src/oracle_resync.rs#L18
• orchestrator/orchestrator/src/ethereum_event_watcher.rs#L23

Polling is very CPU intensive without providing any advantages compared to Event Driven programming.

Recommendation

Currently the Ethereum even watcher works as following,

1. Find the last_cosmos_event_nonce from Cosmos validators.
2. Find the last_block_height_with_last_cosmos_event_nonce from reading old eth logs.
3. Collect logs from last_block_height_with_last_cosmos_event_nonce till last_block_height.
4. Process events from logs and relay to Cosmos if there is any.
5. Update last_block_height_with_last_cosmos_event_nonce with new latest_block_height.
6. Go to (3).

Recomendation 1

It would be nice to come up a way to get back the eth_block_height of the latest_cosmos_event_nonce (GetLastEventNonceByValidator from Cosmos module).
This will remove the need for performance-heavy oracle_sync.rs at step (2).

Maybe something similar to this function SetLastObservedEthereumBlockHeight from Cosmos module?
This is not implemented at Orchestrator side.

Recommendation 2

From a performance point of view, it would be a good idea to switch from loop-based-polling to event streams. For example, geth supports events via pubsub; it supports also fliters for logs.

This will eliminate the need to query in a loop and the CPU heavy mechanisms of loop speed and wait to poll for the next notifications.
It is possible to tokio::spawn a future at each event to keep the current future unblocked for the next event handling.

With these, a new procedure would be as following,

0. Spawn two futures - Fut_1 and Fut_2.
1. (Fut_1) Spawn the pubsub event subscription.
2. (Fut_1) Gather events but don't relay yet. Wait to resync with past events.
3. (Fut_2) Find the last_cosmos_event_nonce and last_block_height_with_last_cosmos_event_nonce from Cosmos validators.
4. (Fut_2) Start reading logs from last_block_height_with_last_cosmos_event_nonce till last_block_height.
5. (Fut_2) Process events from logs and relay to Cosmos if there is any.
6. (Fut_2) Signal pubsub event Future that resync is finished.
7. (Fut_1) Start processing live events.
8. (Fut_1) If pubsub fails, goto (0).