Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@Groxan
Groxan authored Jun 7, 2020
1 parent f460ce2 commit 9ca6583
Showing 1 changed file with 27 additions and 260 deletions.
287 changes: 27 additions & 260 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,65 +1,26 @@
# TzKT (v1-preview)
# Tezos Indexer by Baking Bad
[![Made With](https://img.shields.io/badge/made%20with-C%23-success.svg?)](https://docs.microsoft.com/en-gb/dotnet/csharp/language-reference/)
[![License: MIT](https://img.shields.io/github/license/baking-bad/netezos.svg)](https://opensource.org/licenses/MIT)

A lightweight, API-first, baking-focused Tezos explorer supported by the Tezos Foundation.
TzKT is a lightweight Tezos blockchain indexer with an advanced API created by the [Baking Bad](https://baking-bad.org/docs) team with huge support from the [Tezos Foundation](https://tezos.foundation/).

[API Documentation](https://api.tzkt.io/)
The indexer fetches raw data from the Tezos node, then processes it and stores in the database in such a way as to provide effective access to the blockchain data. For example, getting operations by hash, or getting all operations of the particular account, or getting detailed baking rewards, etc. None of this can be accessed via node RPC, but TzKT indexer makes this data (and much more) available.

### Requirements
## Features:
- **More detailed data.** TzKT not only collects blockchain data, but also processes and extends it with unique properties or even entities. For example, TzKT was the first indexer introduced synthetic operation types such as "migration" or "revelation penalty", which fill in the gaps in account history (because this data is missed in the blockchain), and the only indexer that correctly distinguishes smart contracts among all contracts.
- **Data quality comes first!** You will never see an incorrect account balance, or total rolls, or missed operations, etc. TzKT was built by professionals who know Tezos from A to Z (or, in other words, from TZ to KT 😼).
- **Advanced API.** TzKT provides a REST-like API, so you don't have to connect to the database directly. In addition to basic data access TzKT API has a lot of cool features such as exporting account statements, calculating historical balances (at any block), injecting metadata and much more. See the [API documentation](https://api.tzkt.io), automatically generated using Swagger (Open API 3 specification).
- **Low resource consumption.** TzKT is fairly lightweight. The indexer consumes up to 128MB of RAM, and the API up to 256MB-1024MB, depending on the configured cache size.
- **No local node needed.** TzKT indexer works well even with remote RPC node. By default it uses [tezos.giganode.io](https://tezos.giganode.io/), the most performant public RPC node in Tezos, which is more than enough for most cases.
- **Quick start.** Indexer bootstrap takes ~15 minutes by using snapshots publicly available for all supported networks. Of course, you can run full synchronization from scratch as well.
- **Validation and diagnostics.** TzKT indexer validates all incoming data so you will never get to the wrong chain and will never commit corrupted data. Also, the indexer performs self-diagnostics after each block, which guarantees the correct commiting.
- **Flexibility and scalability.** There is no requirement to run all TzKT components (database, indexer, API) together and on the same machine. This allows flexible optimization, because you can optimize each component separately and according to your needs. Or you can run all the components on the same machine as well, which is much cheaper.
- **PostgreSQL.** TzKT uses the world's most advanced open source database, that gives a lot of possibilities such as removing unused indexes to reduce storage usage or adding specific indexes to increase performance of specific queries. You can configure replication, clustering, partitioning and much more. You can use a lot of plugins to enable cool features like GraphQL. This is a really powerful database.
- **Friendly support.** We are always happy to help everyone and are open to discussions and feature requests. Feel free to [contact us](https://baking-bad.org/docs#contacts).

- .NET Core 3.1
- Postgresql 12
## Installation (docker)

## Tzkt.Sync

This is a flexible and lightweight Tezos blockchain indexer. At the moment it is only lightweight actually, but it's only a matter of time. It uses Postgresql as its main storage via Npgsql and depends on Tzkt.Data, a library with data models and database migrations, that describes the entire database schema using the code-first approach.

The indexer doesn't require a local node with RPC and can be bootstrapped from the snapshot (~900MB), so there is no need to download a huge amount of data.

### RPC calls

In production mode the indexer (if bootstrapped) uses the following RPC endpoints:

- `/chains/main/blocks/{level}`
- `/chains/main/blocks/{level}/context/constants`
- `/chains/main/blocks/{level}/helpers/baking_rights`
- `/chains/main/blocks/{level}/helpers/endorsing_rights`

And there is also a diagnostics option, which does a self test after each block using the following RPC endpoints:

- `chains/main/blocks/{level}/context/raw/json/contracts/global_counter`
- `chains/main/blocks/{level}/context/contracts/{address}`
- `chains/main/blocks/{level}/context/delegates/{address}`

> **Note:** It is highly recommended not to enable diagnostics until the indexer is fully synchronized with the blockchain. Otherwise synchronization will take a long time.
### Costs

At the end of the 173rd cycle in the Tezos mainnet we have the following values:

- **~70MB RAM** (maximum 100MB during full synchronization)
- **~5GB of disk space** for database: 1GB data + 4GB indexes *(optional)*

## Tzkt.Api

This is a native API server for the Tzkt indexer. It actually doesn't depend on the Tzkt.Sync, but on the Tzkt.Data. That means that you can run an API and an indexer separately.

### Documentation

Tzkt.Api uses swagger to automatically generate documentation with Open API 3 specification. The JSON schema is available at `/v1/swagger.json` and can be easily used with open-source third-party UI solutions like Swagger UI, ReDoc, etc.

### Costs

- **~150-400MB RAM**, depending on cache size settings.

## Current state

TzKT (v1-preview) has just been launched and we are actively testing it and are working on adding additional features. At the same time we are working on the frontend part of the TzKT explorer. There are so many things to do!

## Installation guide

The easiest way to try TzKT Indexer and TzKT API is to use Docker. First of all, install `git`, `make`, `docker`, `docker-compose`, then run the following commands:
First of all, install `git`, `make`, `docker`, `docker-compose`, then run the following commands:

````sh
git clone https://github.com/baking-bad/tzkt.git
Expand All @@ -73,9 +34,9 @@ curl http://127.0.0.1:5000/v1/head
make stop
````

## Advanced installation guide
## Installation (from source)

This installation guide is for Ubuntu 18.04 because it is a fairly common OS. If you are using a different OS, the installation process will probably differ only in the "Install packages" step.
This is the preferred way, because you have more control over each TzKT component (database, indexer, API). This guide is for Ubuntu 18.04, but if you are using a different OS, the installation process will probably differ only in the "Install packages" step.

### Install packages

Expand Down Expand Up @@ -167,8 +128,8 @@ Like this:

"TezosNode": {
"ChainId": "NetXdQprcVkpaWU",
"Endpoint": "https://rpc.tzkt.io/mainnet/",
"Timeout": 30
"Endpoint": "https://mainnet-tezos.giganode.io/",
"Timeout": 60
},

"ConnectionStrings": {
Expand Down Expand Up @@ -284,201 +245,7 @@ dotnet Tzkt.Api.dll

That's it. By default API is available on ports 5000 (HTTP) and 5001 (HTTPS). If you want to use HTTPS, you also need to configure certificates. If you want to run API on a different port, add the `"Kestrel"` section to the `appsettings.json` (see example below).

## Install Tzkt Indexer and API for babylonnet

In general the steps are the same as for the mainnet, you just need to use different database, different snapshot and different appsettings (chain id and RPC endpoint). Anyway, let's do it from scratch.

### Prepare database

#### Create an empty database and its user

````
sudo -u postgres psql
postgres=# create database baby_tzkt_db;
postgres=# create user tzkt with encrypted password 'qwerty';
postgres=# grant all privileges on database baby_tzkt_db to tzkt;
postgres=# \q
````

#### Download fresh snapshot

````c
cd ~
wget "https://tzkt-snapshots.s3.eu-central-1.amazonaws.com/baby_tzkt_198.backup" -O baby_tzkt_db.backup
````

#### Restore database from the snapshot

````c
// babylonnet restoring takes ~1 min
sudo -u postgres pg_restore -c --if-exists -v -d baby_tzkt_db -1 baby_tzkt_db.backup
````

### Clone, build, configure and run Tzkt Indexer

#### Clone

````
cd ~
git clone https://github.com/baking-bad/tzkt.git
````

#### Build indexer

````
cd ~/tzkt/Tzkt.Sync/
dotnet publish -o ~/baby-tzkt-sync
````

#### Configure indexer

Edit configuration file `~/baby-tzkt-sync/appsettings.json` with your favorite text editor. What you need is to specify `Diagnostics` (just disable it), `TezosNode.ChainId`, `TezosNode.Endpoint` and `ConnectionStrings.DefaultConnection`.

Like this:

````json
{
"Protocols": {
"Diagnostics": false,
"Validation": true
},

"TezosNode": {
"ChainId": "NetXUdfLh6Gm88t",
"Endpoint": "https://rpc.tzkt.io/babylonnet/",
"Timeout": 30
},

"ConnectionStrings": {
"DefaultConnection": "server=localhost;port=5432;database=baby_tzkt_db;username=tzkt;password=qwerty;"
},

"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
}
}
````

#### Run indexer

````c
cd ~/baby-tzkt-sync
dotnet Tzkt.Sync.dll

// info: Microsoft.Hosting.Lifetime[0]
// Application started. Press Ctrl+C to shut down.
// info: Microsoft.Hosting.Lifetime[0]
// Hosting environment: Production
// info: Microsoft.Hosting.Lifetime[0]
// Content root path: /home/tzkt/baby-tzkt-sync
// warn: Tzkt.Sync.Services.Observer[0]
// Observer is started
// info: Tzkt.Sync.Services.Observer[0]
// Applied 205790
// info: Tzkt.Sync.Services.Observer[0]
// Applied 205791
// ....
````

That's it. If you want to run the indexer as a daemon, take a look at this guide: https://docs.microsoft.com/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-3.1#create-the-service-file.

### Build, configure and run Tzkt API for the babylonnet indexer

Suppose you have already created database `baby_tzkt_db`, database user `tzkt` and cloned Tzkt repo to `~/tzkt`.

#### Build API

````
cd ~/tzkt/Tzkt.Api/
dotnet publish -o ~/baby-tzkt-api
````

#### Configure API

Edit configuration file `~/baby-tzkt-api/appsettings.json` with your favorite text editor. What you need is to specify `ConnectionStrings.DefaultConnection`, a connection string for the database created above.

By default API is available on ports 5000 (HTTP) and 5001 (HTTPS). If you want to use HTTPS, you also need to configure certificates.

If you want to run API on a different port, add the `"Kestrel"` section to the `appsettings.json`.

Like this:

````js
{
"Sync": {
"CheckInterval": 20,
"UpdateInterval": 10
},

"Metadata": {
"AccountsPath": "*"
},

"Cache": {
"LoadRate": 0.75,
"MaxAccounts": 32000
},

"ConnectionStrings": {
"DefaultConnection": "server=localhost;port=5432;database=baby_tzkt_db;username=tzkt;password=qwerty;"
},

"Kestrel": {
"EndPoints": {
"Http": {
"Url": "http://localhost:5010"
},
"Https": {
"Url": "https://localhost:5011"
}
}
},

"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},

"AllowedHosts": "*"
}
````

#### Run API

````c
cd ~/baby-tzkt-api
dotnet Tzkt.Api.dll

// info: Tzkt.Api.Services.Metadata.AccountMetadataService[0]
// Accounts metadata not found
// info: Tzkt.Api.Services.Sync.SyncWorker[0]
// Sync worker initialized with level 205804 and blocks time 30s
// info: Tzkt.Api.Services.Sync.SyncWorker[0]
// Syncronization started
// info: Microsoft.Hosting.Lifetime[0]
// Now listening on: http://localhost:5010
// info: Microsoft.Hosting.Lifetime[0]
// Now listening on: https://localhost:5011
// info: Microsoft.Hosting.Lifetime[0]
// Application started. Press Ctrl+C to shut down.
// info: Microsoft.Hosting.Lifetime[0]
// Hosting environment: Production
// info: Microsoft.Hosting.Lifetime[0]
// Content root path: /home/tzkt/baby-tzkt-api
// ....
````

That's it.

## Install Tzkt Indexer and API for carthagenet
## Install Tzkt Indexer and API for Carthagenet

In general the steps are the same as for the mainnet, you just need to use different database, different snapshot and different appsettings (chain id and RPC endpoint). Anyway, let's do it from scratch.

Expand Down Expand Up @@ -625,10 +392,10 @@ Like this:
"Kestrel": {
"EndPoints": {
"Http": {
"Url": "http://localhost:5020"
"Url": "http://localhost:5010"
},
"Https": {
"Url": "https://localhost:5021"
"Url": "https://localhost:5011"
}
}
},
Expand Down Expand Up @@ -672,7 +439,7 @@ dotnet Tzkt.Api.dll

That's it.

## Install Tzkt Indexer and API for zeronet
## Install Tzkt Indexer and API for Zeronet

In general the steps are the same as for the mainnet, you just need to use different database, different snapshot and different appsettings (chain id and RPC endpoint). Anyway, let's do it from scratch.

Expand All @@ -693,7 +460,7 @@ postgres=# \q

````c
cd ~
wget "https://tzkt-snapshots.s3.eu-central-1.amazonaws.com/zero_tzkt_9535.backup" -O zero_tzkt_db.backup
wget "https://tzkt-snapshots.s3.eu-central-1.amazonaws.com/zero_tzkt_9936.backup" -O zero_tzkt_db.backup
````

#### Restore database from the snapshot
Expand Down Expand Up @@ -819,10 +586,10 @@ Like this:
"Kestrel": {
"EndPoints": {
"Http": {
"Url": "http://localhost:5030"
"Url": "http://localhost:5020"
},
"Https": {
"Url": "https://localhost:5031"
"Url": "https://localhost:5021"
}
}
},
Expand Down

0 comments on commit 9ca6583

Please sign in to comment.