Purge Blocks to a specific height

[dmigwi]

In decred we have a reorganization (or reorg) of the blockchain where a set of blocks are replaced with another set which has more work. The number of blocks replaced is the depth of the reorg. The old set of blocks goes to side chains not available on re-sync.

Can we have an implementation accessible to the bchain.coin package that allows a number of blocks to be dropped?

[martinboehm]

I do not exactly understand what you need.
Blockbook does support reorgs. It detects that it is on wrong chain, removes old blocks from the DB and re-syncs to the new chain.
Please explain more in detail what do you want to achieve.

[dmigwi]

That sounds helpful and I will check the reorg implementation supported.

[martinboehm]

There is a configuration parameter

blockbook/configs/coins/bitcoin.json

Line 57 in 104f6f9

"block_addresses_to_keep": 300,

which specifies the maximum depth of reorg that Blockbook supports, by default 300 blocks. If reorg is deeper, Blockbook will report an error and the whole DB will have to be reindexed.

If you think that there is a chance of deeper reorgs, change it to some higher number for your coin.

[dmigwi]

Hi! @martinboehm, How exactly does the block reorg / purging some unwanted blocks during the sync work?

I tried sending an older block than requested through GetBlock in an effort to remove the unwanted blocks during sync but I got this error:

F0626 05:24:47.817345   21867 sync.go:271] writeBlockWorker skipped block, expected block 80741, new block 80740

My implementation using the reorg concept isn't that great but I wonder if there is a way to drop some blocks and probably reset the height from where the sync should proceed from?

[martinboehm]

You have to return a newest block hash using chain.GetBestBlockHash and then different (then already synchronized) block hash at Blockbook's synchronized height (chain.GetBlockHash). Blockbook then decreases in a loop height by one and compares the hash in DB with hash returned by chain.GetBlockHash until it finds math. All blocks that did not match are removed and synchronization starts again. See the code here

blockbook/db/sync.go

Lines 87 to 110 in b1810dc

	remoteBestHash, err := w.chain.GetBestBlockHash()
	if err != nil {
	return err
	}
	localBestHeight, localBestHash, err := w.db.GetBestBlock()
	if err != nil {
	return err
	}
	// If the locally indexed block is the same as the best block on the network, we're done.
	if localBestHash == remoteBestHash {
	glog.Infof("resync: synced at %d %s", localBestHeight, localBestHash)
	return errSynced
	}
	if localBestHash != "" {
	remoteHash, err := w.chain.GetBlockHash(localBestHeight)
	// for some coins (eth) remote can be at lower best height after rollback
	if err != nil && err != bchain.ErrBlockNotFound {
	return err
	}
	if remoteHash != localBestHash {
	// forked - the remote hash differs from the local hash at the same height
	glog.Info("resync: local is forked at height ", localBestHeight, ", local hash ", localBestHash, ", remote hash", remoteHash)
	return w.handleFork(localBestHeight, localBestHash, onNewBlock, initialSync)
	}

and the content of the handleFork method.

[dmigwi]

This reorg implementation seems to work only before the first sync runs, Is there a way to do it while the first sync is in progress?

[dmigwi]

This is the case scenario that I am trying to fix.

The proof of stake implementation which could benefit from the reorg during sync works like this:

When a block is mined on decred block chain as the best block, it doesn't have the confirming votes cast to validate it. The next consecutive block mined either validates or invalidates that block whereby the txs in the invalidated block are mined in other following blocks.

I was hopeful that a reorg implementation during sync would help me address that unique concern on decred's part since I don't have access to the db so as to drop/delete txs in the invalidated block.

[dmigwi]

Also does one have to restart the code for reorg to work?

[martinboehm]

This reorg implementation seems to work only before the first sync runs, Is there a way to do it while the first sync is in progress?

On the contrary, during the inital sync we do not expect any reorg as the initial sync is starting with old blocks with many confirmations. Only in the end, close to the chain top there can be a reorg.

When a block is mined on decred block chain as the best block, it doesn't have the confirming votes cast to validate it. The next consecutive block mined either validates or invalidates that block whereby the txs in the invalidated block are mined in other following blocks.

I do not really see any difference to let's say Bitcoin but I probably do not understand the decred implementation enough.

Also does one have to restart the code for reorg to work?

No, you do not have to restart.

[dmigwi]

On the contrary, during the inital sync we do not expect any reorg as the initial sync is starting with old blocks with many confirmations. Only in the end, close to the chain top there can be a reorg.

ok. Noted. Is there a way that I could remove the transaction details of a synced invalidated block that falls within set the max reorg depth during the first sync?

[dmigwi]

I wish there was a function similar to this code, accessible to the coins package which could help me delete the unwanted transactions in the invalidated block during the first sync. I am experiencing some bugs due to the hacks I have resorted to in order to avoid transactions in invalidated blocks from being saved in the rocksdb.

blockbook/db/rocksdb.go

Lines 1282 to 1340 in b1810dc

	// DisconnectBlockRangeBitcoinType removes all data belonging to blocks in range lower-higher
	// it is able to disconnect only blocks for which there are data in the blockTxs column
	func (d *RocksDB) DisconnectBlockRangeBitcoinType(lower uint32, higher uint32) error {
	blocks := make([][]blockTxs, higher-lower+1)
	for height := lower; height <= higher; height++ {
	blockTxs, err := d.getBlockTxs(height)
	if err != nil {
	return err
	}
	if len(blockTxs) == 0 {
	return errors.Errorf("Cannot disconnect blocks with height %v and lower. It is necessary to rebuild index.", height)
	}
	blocks[height-lower] = blockTxs
	}
	wb := gorocksdb.NewWriteBatch()
	defer wb.Destroy()
	txAddressesToUpdate := make(map[string]*TxAddresses)
	txsToDelete := make(map[string]struct{})
	balances := make(map[string]*AddrBalance)
	for height := higher; height >= lower; height-- {
	blockTxs := blocks[height-lower]
	glog.Info("Disconnecting block ", height, " containing ", len(blockTxs), " transactions")
	// go backwards to avoid interim negative balance
	// when connecting block, amount is first in tx on the output side, then in another tx on the input side
	// when disconnecting, it must be done backwards
	for i := len(blockTxs) - 1; i >= 0; i-- {
	btxID := blockTxs[i].btxID
	s := string(btxID)
	txsToDelete[s] = struct{}{}
	txa, err := d.getTxAddresses(btxID)
	if err != nil {
	return err
	}
	if txa == nil {
	ut, _ := d.chainParser.UnpackTxid(btxID)
	glog.Warning("TxAddress for txid ", ut, " not found")
	continue
	}
	if err := d.disconnectTxAddresses(wb, height, btxID, blockTxs[i].inputs, txa, txAddressesToUpdate, balances); err != nil {
	return err
	}
	}
	key := packUint(height)
	wb.DeleteCF(d.cfh[cfBlockTxs], key)
	wb.DeleteCF(d.cfh[cfHeight], key)
	}
	d.storeTxAddresses(wb, txAddressesToUpdate)
	d.storeBalancesDisconnect(wb, balances)
	for s := range txsToDelete {
	b := []byte(s)
	wb.DeleteCF(d.cfh[cfTransactions], b)
	wb.DeleteCF(d.cfh[cfTxAddresses], b)
	}
	err := d.db.Write(d.wo, wb)
	if err == nil {
	glog.Infof("rocksdb: blocks %d-%d disconnected", lower, higher)
	}
	return err
	}