Proposal: Swarm DB - Data Pipelilnes

[molekilla]

Swarm DB Data Pipelines

Swarm DB Data Pipelines for long-running, intensive storage tasks

Summary

Imagine a "data dex", where you ask the dex "Move up to 1 MB of Filecoin data to Swarm and update cid references", and a set of smart contracts, routers, oracles and bridges gives a single quote and estimation of a work order. Here we define how this is a potential paradigm and revolutionary idea when used in conjuction with the forthcoming Swarm DB.

Guide-level explanation

Similar to ZigZag Arweave Bridge, Swarm DB Data Pipelines is agnostic in nature, where you choose the data storage provider from and to and the fungible token to use for payment.

The actors in this use case are the following:

Dex (Decentralized Exchange)

A set of dexes which maintains an Automated Market Maker with support for the decentralized storage chain token.

Eg. Quote 100 DAI of BZZ

Storage Allocation contract

A set of oracles or smart contracts to query storage allocation.

Eg Quote 1MB of storage in Filecoin and Arweave

Swarm DB

Part of Swarm DB features, sends a query for job / work orders computation estimates and execute them.

Eg. Quote 256 MB memory, 1 GB bandwidth and 20 CPU shares*

*=WASM engine must be able to calculate based on memory, CPU and bandwidth.

Destination chain

A chain which is compatible with multiformats / IPLD.

Routers and/or bridges

Technical Implementation

We have an use case where we need to stored existing data in Filecoin, just a portion of it, and that it maintains linkability, that is, the set of CARs/CIDs copied to Swarm or any other IPLD-compatible bridge, can be reference accordingly or traverse.

An implementation that satisfies this use case requires:

Select Filecoin CIDs to move

User selects a set of CIDs and CAR is created and stored in Swarm.

CAR is send to a Swarm DB Job Work Order for Quotation

A computation query is required for cost planning.

CAR is send to a Filecoin Job Work Order for Quotation

A computation query is required for cost planning (to update cid references)

Dapp receives cost planning and quotes in selected fungible token for dex prices

Costs are display in user's selected fungible token

Dapp sends cost planning quote to Swarm DB router network for availability

A router asks register Swarm DB instances for quotation of work orders

Dapp approves and send work orders to Swarm DB node

One or more work orders, are sent to the available Swarm DB Node.

Storage allocation is purchased, and computation is executed

Required postage stamp batch IDs, tokens and gas is purchased

CAR is send to work order node for computation

Node fetches CAR file and converts it to beeson, which then are stored in bee including inclusion proofs.

Once complete or rejected, a notification is sent and emitted by a smart contract

Dapp is notified of changes.

If succesful, a claim must be executed to close the work order, or retry/refund

To finish work order, user claims work order and a signature is attached to the work order which certifies acknowledge by the
user.

Glossary

• CAR: Content Addressable aRchive
• CID: Content IDentifier
• DEX: Decentralized Exchange
• AMM: Automated Market Maker
• IPFS: Interplanetary File System
• Swarm: Ethereum Swarm Decentralized Content Storage Network
• DApp: Decentralize Application
• IPLD: Interplanetary Linked Data
• Filecoin: Proof of Storage and Space decentralized content storage chain
• WASM: Web Assembly
• Swarm DB: Decentralized database using Swarm Bee as store and a set of smart contracts and L2 Nodes which offer a database-like experience

Copyright

Copyright and related rights waived via CC0.

Author

@molekilla (Rogelio Morrell)