deterministic snapshots #5956

wanderingbort · 2018-10-10T16:31:02Z

Starting this review early.

This is the plumbing and unit tests for creating:

deterministic platform agnostic snapshots of all of the consensus state required to bootstrap a node at a given head block
- these snapshots are taken of the HEAD state of the chain BUT are loaded as if irreversible so, snapshots should only be used if the head block they represent become irreversible until irreversible read mode is available.
- these snapshots can be loaded optionally replaying blocks from a block.log if present and newer than the snapshot
- the node may continue on perpetually but cannot act as a source of syncing for blocks which are not present in the block.log (it is valid to provide a full block log and a snapshot to just skip the time it takes to build up the state of the chain)
- the snapshot can be binary or JSON encoded (but its really quite large most of the time)
a deterministic platform agnostic integrity hash of the entire chain state (comparable between nodes to check derived state).

Things left to do:

RPC command to drop the next convenient snapshot
application level versioning
net_plugin review to make sure incomplete block logs don't break anything critical

instructions for using snapshots

configuring a snapshot directory

By default snapshots are written to the snapshots directory relative to your nodeos data directory. This can be overridden with a relative or absolute path using the snapshots_dir configuration or using the --snapshots_dir command line option.

Snapshots are written out to files named with the pattern snapshot-<head block id in hex>.bin

Triggering the creation of a snapshot

Snapshots can be triggered at runtime using the RPC available through the producer_api_plugin. For example:

$ curl http://127.0.0.1:8888/v1/producer/create_snapshot | json_pp
{
   "snapshot_name" : "<data-dir>/snapshots/snapshot-0000002091fd7cb9f1656d25d3a20a93c8aff044c5de19d7cf6f9bab094e3a56.bin",
   "head_block_id" : "0000002091fd7cb9f1656d25d3a20a93c8aff044c5de19d7cf6f9bab094e3a56"
}

Instantiating a node from a snapshot

When starting a new instance of nodeos the --snapshot <path to snapshot> command line option can be used instead of replaying a blocks.log or resyncing from a network. If a blocks.log is provided it MUST at least contain blocks up to the snapshotted block and MAY contain additional blocks that will be applied as part of startup. In addition, any available reversible/ blocks will be applied.

When instantiating a node from a snapshot, it is illegal to pass in any --genesis-* arguments as that information is loaded from a snapshot. If a blocks.log exists, the genesis information it contains will be validated against the genesis data in the snapshot and error if they are not consistent.

NOTE instantiating a node from a snapshot without a blocks.log is valid BUT it will create a partial blocks.log which will affect its ability to service RPC/P2P requests for block data.

Instructions for using integrity hashes

Integrity hashes are a way of comparing the contents of the blockchain state database. They consist of a sha256 hash of a deterministic binary representation of all consensus affecting state.

calculating an integrity hash

Integrity hashes can be retrieved at runtime using the RPC available through the producer_api_plugin. For example:

$ curl http://127.0.0.1:8888/v1/producer/get_integrity_hash | json_pp
{
   "head_block_id" : "00000503289f5f9acd8a8f880b6d91d09817e6806e8f428c6081634a99f3e923",
   "integrity_hash" : "f6c9d3d088e6ceaf87af5e52f8b210a78caf8b831c3b8e890c96c0e6edf734c9"
}

This reverts commit e2ec9f9.

…tic-snapshot

…x several bugs that result from that concept

…rom a snapshot, fix bugs that bubbled up as a result

…tation details in source files or detail namespaces

…tic-snapshot

…m between header info and blocks

…or return the integrity hash over the database

…e a node from a snapshot

wanderingbort · 2018-10-10T23:01:24Z

@dskvr please see the documentation above and comment on any concerns 😄

heifner · 2018-10-11T13:05:24Z

libraries/chain/block_log.cpp

+      my->block_stream.write((char*)&totem, sizeof(totem));
+
+      if (first_block) {
+         auto ret = append(first_block);


I would think ret would be warning with some compilers.. unused.

…e irreversible block implied by the snapshot

…optimized for undo

* contract tables are now in their own index set * this index set is snapshotted and included in the integrity check through a different process * traversal of the normal index set will include the table_id_objects * traversal of the contracts tables is then free to use those objects to traverse tables in the order of the table_id_objects * this will traverse logical tables instead of the whole combined table * each logical table gets its own section and is ordered by the next most appropriate key depending on its type

…ake up space and are not worth it

wanderingbort · 2018-10-15T15:46:19Z

The original version of the snapshot was not safe WRT foreign keys, The second version of the snapshot was really slow. This represents a 3rd attempt at foreign key support that is performant. Smoke tests on existing chains are coming back positive.

wanderingbort · 2018-10-15T16:01:45Z

depends on EOSIO/fc#29

…napshot datafile and the structure of the snapshotted data are versioned separately

…tic-snapshot

ChenJiaoZhu · 2019-01-21T07:13:26Z

the snapshot can be binary or JSON encoded

Could you please tell me how to create the snapshot in JSON format? Thanks

b1bart added 27 commits August 24, 2018 09:37

add some basic utilities to help consistently work with the database

9d8896b

add authorization and resource management to integrity hash

ed320c0

add accessor for integrity hash

a0681e6

initial plumbing for snapshot

3a06684

fix some issues

c783578

move code, abi and packed trx to shared_blob which jsons to base 64

edbcd13

move kv values to shared_blob

9e83384

add resource and authorization managers to snapshot

a8c7da0

allow sink to decide between variant or fc::raw::packed forms

2800c2e

add writer interface and prototype to JSON

5318041

add signals for plugins

e2ec9f9

Revert "add signals for plugins"

89cdf62

This reverts commit e2ec9f9.

plumbing for loading/restoring from snapshot

94510b6

beginnings of unit test for snapshot

013acac

Merge remote-tracking branch 'origin/develop' into feature/determinis…

0312aa1

…tic-snapshot

restored fc changes lost in merge

a4f7f85

add basic snapshot test, modify block log to support partial logs, fi…

cdbe373

…x several bugs that result from that concept

add contract state manipulation to the test, add test for replaying f…

0d3f264

…rom a snapshot, fix bugs that bubbled up as a result

add validation to the snapshot reader abstraction, hide more implemen…

51df148

…tation details in source files or detail namespaces

add binary snapshot and tests

e4f4cb6

fix for mutable_variants not being aliasable to variant

aa2a693

fixes to binary serialization caused by overloaded operators

45bdd37

Merge remote-tracking branch 'origin/develop' into feature/determinis…

da7858c

…tic-snapshot

fix irreversible read mode test

ca1b970

fix block_log::repair_log which was missing the new separating tote…

d7a6ed7

…m between header info and blocks

add RPC in producer_plugin_api to trigger the creation of a snapshot …

730f253

…or return the integrity hash over the database

add genesis information to snapshot, add snapshot option to initializ…

d0fb380

…e a node from a snapshot

initialize first_block_num for legacy blocks.log files

9ef0e13

heifner reviewed Oct 11, 2018

View reviewed changes

b1bart added 3 commits October 11, 2018 09:42

validate that the block log matches or exceeds the snapshot, track th…

7fb374c

…e irreversible block implied by the snapshot

no variadic max duh

e633866

work on fixing the consistency problems with foreign keys and tables …

86f3dd6

…optimized for undo

heifner mentioned this pull request Oct 12, 2018

Snapshot plugin #4837

Closed

b1bart added 5 commits October 12, 2018 16:41

protect against future attempts to snapshot internal chainbase OIDs

390f093

remove empty contract table sections from the snapshot as they just t…

0fc2531

…ake up space and are not worth it

interim attempt at another direction

ef7bd03

fix a few bugs with empty sections and empty table types

ed3df78

b1bart added 4 commits October 15, 2018 16:38

bump libraries/fc to master with changes

10ca4e6

add basic application level versioning so that the structure of the s…

de4b8d7

…napshot datafile and the structure of the snapshotted data are versioned separately

remove unused variable

7d750e1

Merge remote-tracking branch 'origin/develop' into feature/determinis…

5b84e24

…tic-snapshot

wanderingbort added this to the Version 1.4 milestone Oct 16, 2018

Merge remote-tracking branch 'origin/develop' into feature/determinis…

9b37cb8

…tic-snapshot

wanderingbort changed the title ~~[WIP] deterministic snapshots~~ deterministic snapshots Oct 16, 2018

wanderingbort merged commit 1520889 into develop Oct 16, 2018

wanderingbort deleted the feature/deterministic-snapshot branch October 16, 2018 21:36

Yucheng123 mentioned this pull request Dec 27, 2018

database dirty flag set (likely due to unclean shutdown): replay required #4742

Closed

ChenJiaoZhu mentioned this pull request Jan 22, 2019

How to create the snapshot in JSON format? #6649

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deterministic snapshots #5956

deterministic snapshots #5956

wanderingbort commented Oct 10, 2018 •

edited by knmorgan

Loading

wanderingbort commented Oct 10, 2018

heifner Oct 11, 2018

wanderingbort commented Oct 15, 2018

wanderingbort commented Oct 15, 2018

ChenJiaoZhu commented Jan 21, 2019 •

edited

Loading

deterministic snapshots #5956

deterministic snapshots #5956

Conversation

wanderingbort commented Oct 10, 2018 • edited by knmorgan Loading

instructions for using snapshots

configuring a snapshot directory

Triggering the creation of a snapshot

Instantiating a node from a snapshot

Instructions for using integrity hashes

calculating an integrity hash

wanderingbort commented Oct 10, 2018

heifner Oct 11, 2018

Choose a reason for hiding this comment

wanderingbort commented Oct 15, 2018

wanderingbort commented Oct 15, 2018

ChenJiaoZhu commented Jan 21, 2019 • edited Loading

wanderingbort commented Oct 10, 2018 •

edited by knmorgan

Loading

ChenJiaoZhu commented Jan 21, 2019 •

edited

Loading