Hedera Mirror Node exposes Hedera Hashgraph transactions, transaction records, account balances, and events generated by the Hedera mainnet (or testnet, if so configured) via a REST & gRPC API.
Hedera mirror nodes receive the information from the mainnet nodes and they provide value-added services such as providing audit support, access to historical data, transaction analytics, visibility services, security threat modeling, data monetization services, etc. Mirror nodes can also run additional business logic to support applications built using Hedera mainnet.
While mirror nodes receive information from the mainnet nodes, they do not contribute to consensus on the mainnet, and their votes are not counted. Only the votes from the mainnet nodes are counted for determining consensus. The trust of Hedera mainnet is derived based on the the consensus reached by the mainnet nodes. That trust is transferred to the mirror nodes using cryptographic signatures on a chain of records (account balances, events, transactions, etc).
Eventually, the mirror nodes can run the same code as the Hedera mainnet nodes so that they can see the transactions in real time. To make the initial deployments easier, the beta mirror node strives to take away the burden of running a full Hedera node through creation of periodic files that contain processed information (such as account balances or transaction records), and have the full trust of the Hedera mainnet nodes. The beta mirror node software reduces the processing burden by receiving pre-constructed files from the mainnet, validating those, populating a database and providing REST APIs.
- Lower compute, bandwidth requirement
- It allows users to only save what they care about, and discard what they don’t (lower storage requirement)
- Easy searchable database so the users can add value quickly
- Easy to consume REST APIs to make integrations faster
The beta mirror node works as follows:
-
When a transaction reaches consensus, Hedera nodes add the transaction and its associated record to a record file.
-
The file is closed on a regular cadence and a new file is created for the next batch of transactions and records. The interval is currently set to 5 seconds but may vary between networks.
-
Once the file is closed, nodes generate a signature file which contains the signature generated by the node for the record file.
-
Record files also contain the hash of the previous record file, thus creating an unbreakable validation chain.
-
The signature and record files are then uploaded from the nodes to Amazon S3 and Google File Storage.
-
This mirror node software downloads signature files from either S3 or Google File Storage.
-
The signature files are validated to ensure at least 1/3 of the nodes in the address book (stored in a
0.0.102file) have the same signature. -
For each valid signature file, the corresponding record file is then downloaded from the cloud.
-
Record files can then be processed and transactions and records processed for long term storage.
-
Event files are handled in same manner as record files.
-
In addition, nodes regularly generate a balance file which contains the list of Hedera accounts and their corresponding balance which is also uploaded to S3 and Google File Storage.
-
The files are also signed by the nodes.
-
This mirror node software can download the balance files, validate at least 1/3 of nodes have signed and then process the balance files for long term storage.
Multiple technologies are utilized in the mirror node. The following topics are areas where basic knowledge is valuable to understanding the mirror node operations.
Ensure these tools are installed prior to running the mirror node
To run the mirror node, execute these 3 commands in your terminal.
git clone https://github.com/hashgraph/hedera-mirror-node.git
cd hedera-mirror-node
docker-compose upNOTE: This defaults to a bucket setup for demonstration purposes. The real bucket name is not currently publicly available. See Testnet / Mainnet section for how to configure for real data
The free option utilizes a bucket setup for demonstration purposes. The real bucket name is not currently publicly available. This is the default option and requires no additional steps.
To access real data from the testnet or mainnet buckets the requester pays flow for GCP or AWS must be utilized. Here the charges associated with request and download of transaction data are paid for by the requester and not the bucket owner.
To achieve this, 2 simple steps must be taken to configure the mirror node
- Uncomment the contents of application.yml file under the root folder
- Customize the contents according to your setup. See configurations for detailed descriptions of config options.
Note The application.yml file contents represent the minimal set of fields required to configure requester pays and must all be uncommented and filled in. The file is referenced in the docker-compose.yml and allows customized configuration for each of the sub modules.
See the Docker compose Startup section for further details
When running the Mirror Node using docker, activity logs and container status for each module container can be viewed in docker desktop Dashboard to verify expected operation.
You can also interact with some module containers (gRPC and REST API's) to verify their operation.
First list running docker container information using
docker ps
Useful information for all the running containers such as CONTAINER ID, STATUS, IP's and ports will be displayed as below
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21fa2a986d99 gcr.io/mirrornode/hedera-mirror-rest:0.12.0 "docker-entrypoint.s…" 7 minutes ago Up 12 seconds 0.0.0.0:5551->5551/tcp hedera-mirror-node_rest_1
56647c384d49 gcr.io/mirrornode/hedera-mirror-grpc:0.12.0 "java -cp /app/resou…" 8 minutes ago Up 16 seconds 0.0.0.0:5600->5600/tcp edera-mirror-node_grpc_1
An 'Up' STATUS confirms the mirror node sub modules are running.
Using the IP and port, the gRPC and REST API's endpoints can be called to confirm data is processed and available.
Using the CONTAINER ID docker containers can be accessed for in depth troubleshooting.
Logs similar to the following snippets can be used to confirm the Importer is downloading and parsing transactions to the database
Current version of schema "public": << Empty Schema >>
...
Migrating schema "public" to version 1.0 - Init
...
Successfully applied 41 migrations to schema "public" (execution time 00:01.656s)
...
Loading record format version 2 from record file: ...
Inserted 2 transactions, 6 transfer lists, 0 files, 0 contracts, 0 claims, 0 topic messages, 0 non-fee transfers
Finished parsing 2 transactions from record file ....
The following log can be used to confirm the database is up and running
LOG: database system is ready to accept connections
If you have postgresql installed you can connect directly to your database using the following psql command and default configurations
psql "dbname=mirror_node host=localhost user=mirror_node password=mirror_node_pass port=5432"
If psql is not available you can docker exec -it <CONTAINER ID> bash into the db container and run the same command above
Some useful basic queries to help view database contents and data include
\du
select * from account_balance limit 5;
select * from transactions limit 5;
select * from topic_message limit 5;
The REST API container will display logs similar to the below at start
Server running on port: 5551
The REST API endpoints can be verified either through the browser or the terminal. The following endpoints are suggestions that can be accessed from your browser. Modify the below IP's and ports if they differ from your running containers.
http://127.0.0.1:5551/api/v1/accounts
http://127.0.0.1:5551/api/v1/balances
http://127.0.0.1:5551/api/v1/transactions
When using the terminal simply use the curl command on the above endpoints. e.g.
curl http://127.0.0.1:5551/api/v1/transactions
The gRPC container will display logs similar to the below at start
MirrorGrpcApplication Started MirrorGrpcApplication ....
Listener Starting to poll every 1000ms
The gRPC streaming endpoint can be verified using clients that support HTTP/2 Some useful clients we're encountered include
-
-
run the following command making substitutions for {topicNum}, {grpcContainerIP} and {grpcContainerPort} appropriately
grpcurl -plaintext -d '{"topicID":{"shardNum":0,"realmNum":0,"topicNum":{topicNum}},"consensusStartTime":{"seconds":0,"nanos":0},"limit":10}' {grpcContainerIP}:{grpcContainerPort} com.hedera.mirror.api.proto.ConsensusService/subscribeTopic
-
Additionally logs on each module container can be viewed to verify expected operation or decipher issues - See Troubleshooting.
Simply access the terminal on each container with the following command and refer to Operations document for directions on where and how to view logs
docker exec -it <CONTAINER ID> bash
Note You cannot exec into the gRPC and Importer containers as they are distroless java images.
To prepare for a new release:
./mvnw clean package -N -P=release -Drelease.version=x.y.z -Drelease.chartVersion=x.y.z
Contributions are welcome. Please see the contributing guide to see how you can get involved.
This project is governed by the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to [email protected]