Indexer S3 data cache to PostgresDB or RocksDB and Redis. PostgresDB or RocksDB cache Indexer Raw. Redis cache state
Get S3 data and tracking new block. Based NEAR Lake.
Http Client post S3 data and state event to url path.
Http Server API to use cache and S3 data. Provide RESTful API and GraphQL.
Get HTTP_SERVER_LISTEN/cache current sync height block data.
Get HTTP_SERVER_LISTEN/cache/<id> custom height block data.
- Local cache use RocksDB
- DB fail Retry
- From Cache
- Network error retry
- Websocket push (graphql-engine)
- Add more api (graphql-engine)
- GraphQL (graphql-engine)
In order to be able to get objects from the AWS S3 bucket you need to provide the AWS credentials.
AWS default profile configuration with aws configure looks similar to the following:
~/.aws/credentials
[default]
aws_access_key_id=
aws_secret_access_key=
AWS docs: Configuration and credential file settings
// This is S3 config
S3_BUCKET_NAME=near-lake-data-mainnet
S3_REGION_NAME=eu-central-1
// Start from cache if true ignore START_BLOCK_HEIGHT
START_BLOCK_HEIGHT_FROM_CACHE=true
START_BLOCK_HEIGHT=0
// PUSH block and event to Engine if false ignore PUSH_ENGINE_URL
ENABLE_PUSH_SERVER=false
PUSH_URL=http://127.0.0.1:8088
// Http Server
HTTP_SERVER_LISTEN=127.0.0.1:8088
// Cache DB
DATABASE_URL=postgres://nearscan_raw:[email protected]/nearscan
cargo build --release
./target/release/octopus-near-indexer-s3