Lens Substream

A Rusty way to stream Lens data using Substreams.

---

Decentralized social media generates a huge amount of on-chain data. This on-chain data needs a low-latency interface for it to remain accessible for all, along with adding the ability to perform real-time analysis and monitoring.

Substreams are a data solution developed for The Graph Network. They allow developers to write Rust modules composing data streams with low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.

This project builds a (basic) Substream in Rust to monitor on-chain Lens protocol data and provide a low-latency data streaming interface. Potential uses of such a tool could be for real-time analysis and monitoring of Lens data. Since Substreams are streamed, the data can be directly injected into a variety of databases (BigQuery, Clickhouse, Kafka etc.), opening up a lot of distinct analytics use cases.

A few unrelated citations:

1. Gizmodo: Here Are 30 of the Hundreds of Subreddits Going Dark to Protest Reddit's API Changes
2. Verge: Twitter just closed the book on academic research
3. 9to5Google: Reddit’s API changes kill Android’s best third-party apps, including RIF and Sync

Installation

1. Copy this repo
2. Get an API key from StreamingFast
3. Install the Substreams CLI
4. Add the API key to your environment variables

export STREAMINGFAST_KEY='xxx'
export SUBSTREAMS_API_TOKEN=$(curl https://auth.streamingfast.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token)

Using Makefile defaults

make codegen
make build
make stream

Manual compilation

Generate protobuf

substreams protogen ./substreams.yaml --exclude-paths="sf/substreams,google"

Compile

cargo build --target wasm32-unknown-unknown --release

Start Substream

substreams run -e polygon.streamingfast.io:443 substreams.yaml map_transfers -s 43764445 -t 100

Limitations

• Tuples are not currently supported by rust-ethereum/ethabi ( see open issue), so they have been removed from the Lens Protocol Profiles (LPP) ABI. Currently, only implements protobuf interfaces for Lens collects and mirrors.