Handle unsynced deployers

[bakhshandeh]

Our system relies on the assumption that every deployer node has access to the complete and updated data. However, this assumption may be violated if a new deployer joins the network or an existing deployer goes offline and misses some updates. In such cases, the deployer's database may be out of sync and may return inaccurate data to the callers

[bakhshandeh]

How to implement?

1- A new plugin (deployers-sync or tier3-sync) that runs only on the deployers. It can be disabled on other nodes or it can run but do nothing.

2- A new table (Settings) will be added to the persistent database (MongoDB) to allow the module to save the timestamp of the latest context in the database.

{
'deployers-sync.last_timestamp': 1234
}

Other necessary settings also can be saved into this table by other plugins.

3- The plugin calls an endpoint or a lip2p protocol on a few random deployers and passes the latest timestamp to see if there are any new contexts that need to be saved.

4- If there is any new context, they will be saved and `Settings.deployers-sync.last_timestamp' will be updated

Notes

• In some cases, a node needs to recover its database from scratch. deployers-sync.last_timestamp is null and the plugin should request data from the beginning. We should add pagination and let the node sync, for example, the first 100 contexts first and update deployers-sync.last_timestamp. Then another page will be requested based on deployers-sync.last_timestamp, and so on.

• This plugin could be merged with db-sync if there are many redundant parts.

• A single or a few dishonest deployers should not be able to send incorrect data to other nodes.

[sadeghte]

Sometimes, a node may have the latest context but not some of the previous ones. This can happen if a node is offline for a period and then comes back online. If a new deployment occurs before the node recovery process start, none of the previous contexts will be restored.

[bakhshandeh]

a new deployment occurs before the node recovery process start, none of the previous contexts will be restored.

We do not update deployers-sync.last_timestamp on deployment. It will be updated only when deployers-sync runs.