[CX-CLEANUP] - DA Integrated Storage

[bfish713]

What is this task and why do we need to work on it?

We want to integrate a persistent storage interface within HotShot to enable the application to manage data storage effectively. This feature aims to address potential data storage issues (e.g., full disk space) that can prevent nodes from storing DA blobs or VID shares.

See details here:
https://www.notion.so/espressosys/DA-integrated-storage-27cba2adb0f744fc8ac99591140627c2

Also Discussion on Zulip:

Jeb Bearer: A while ago we discussed an idea of having a persistent storage interface in HotShot implemented by the application. This would allow the application to store data before HotShot signed a DA proposal. We decided this was not a big enough improvement to bring into decaf because the odds of a node crashing between signing a DA proposal and decide, and thus losing data, is so small.

I want to revisit this decision because there is another failure mode. What if the node doesn't crash, remains online but unable to store data (for example, its disk is full). This condition could persist for some time and thus make it more likely that it happens in the critical window. This becomes more likely when we implement pruning for DA-but-not-archival nodes, if we are having higher volume than expected, the storage requirement for DA retention could exceed our minimum hardware recommendations.

Having an application persistence interface integrated with the DA replica logic would allow the application to stop its node from voting on DA (and similarly VID) if it is unable to store its DA blob (or VID share) for any reason.

Jeb Bearer: Does this sound valuable enough to integrate for decaf?

Brendon Fish: To me this seems reasonable to add, I think. Does the sequencer currently use/impl the Storage trait in Hotshot? Maybe it calls store_leaf on ConsensusAPI? I'm guessing it doesn't but if it does we can probably work from that.

Let me know if my understanding of what Hotshot would need to add is correct:

Create a trait for block storage which the application implements, just like the other Block types
Take/create a storage when initializing hotshot
When we receive a DA proposal, call some append or store function from this trait.
If storing succeeds vote on the DA proposal
If it fails don't vote
Same logic for VID shares except don't vote on the Quorum proposal
Jeb Bearer: Yeah exactly

Jeb Bearer: We could also then remove the in-memory payload/VID stores in HotShot, and remove them from the Decide event. The application could be responsible for finding the relevant data and archiving it when it sees a decide, and garbage collecting it

Jeb Bearer: And on the application side, this is a pretty straightforward key-value store, should be easy to implement in postgres

What work will need to be done to complete this task?

1. Create BlockStorage Trait
2. Initialize BlockStorage in HotShot's init
3. Add Storage Operations on DA Proposals
   • If this fails, do not vote on the proposal and exit the event handler.
4. Add Storage Operations on DA Proposals VID Shares
   • If this fails, do not vote on the VID disperse, otherwise, use the old logic to vote.

Are there any other details to include?

No response

What are the acceptance criteria to close this issue?

• Provide an interface to store DA data, and properly store it using that interface.
• Write a unit test interface for BlockStorage called TestBlockStorage which uses some simple data type for testing purposes.
• Write tests for both the DA and Consensus tasks to validate that data is indeed stored properly.

Branch work will be merged to (if not the default branch)

No response