Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC31: Add a variable length field in the block header #224

Merged
merged 6 commits into from
Apr 1, 2022
Merged

RFC31: Add a variable length field in the block header #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
18b45d3
RFC31: Add a variable length field in the block header
doitian Feb 7, 2021
9c86db3
add test vectors for block hash with extension
doitian Aug 12, 2021
2e737d8
apply suggestions from https://github.com/nervosnetwork/rfcs/pull/224…
doitian Aug 18, 2021
30fd31c
apply suggestions from https://github.com/nervosnetwork/rfcs/pull/224…
doitian Aug 18, 2021
155749b
Apply suggestions from code review
doitian Mar 2, 2022
7da4dea
revise rfc31 according to the review comment
doitian Mar 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion rfcs/0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,11 @@ Similar to NC, in our protocol, a compact block replaces a block’s commitment

Additional block structure rules:

- The total size of the first four fields should be no larger than the hard-coded **block size limit**. The main purpose of implementing a block size limit is to avoid overloading public nodes' bandwidth. The uncle blocks’ proposal zones do not count in the limit as they are usually already synchronized when the block is mined.
- The total size of the first four fields should be no larger than the hard-coded **block size limit**. The main purpose of implementing a block size limit is to avoid overloading public nodes' bandwidth. The uncle blocks’ proposal zones do not count in the limit as they are usually already synchronized when the block is mined. Since [RFC31], the new field `extension` is also counted in the total size.
- The number of `txpid`s in a proposal zone also has a hard-coded upper bound.

[RFC31]: ../0031-variable-length-header-field/0031-variable-length-header-field.md

Two heuristic requirements may help practitioners choose the parameters. First, the upper bound number of `txpid`s in a proposal zone should be no smaller than the maximum number of committed transactions in a block, so that even if *w<sub>close</sub>=w<sub>far</sub>*, this bound is not the protocol's throughput bottleneck. Second, ideally the compact block should be no bigger than 80KB. According to [a 2016 study by Croman et al.](https://fc16.ifca.ai/bitcoin/papers/CDE+16.pdf), messages no larger than 80KB have similar propagation latency in the Bitcoin network; larger messages propagate slower as the network throughput becomes the bottleneck. This number may change as the network condition improves.

#### Block Propagation Protocol
Expand Down
371 changes: 370 additions & 1 deletion rfcs/0031-variable-length-header-field/0031-variable-length-header-field.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,370 @@
In review, see <https://github.com/nervosnetwork/rfcs/pull/224>
---
Number: "0031"
Category: Consensus (Hard Fork)
Status: Draft
Author: Ian Yang
Organization: Nervos Foundation
Created: 2021-02-07
---

# Add a variable length field in the block

## Abstract

This document proposes adding an optional variable length field to the block.

## Motivation

In the consensus version before activating this RFC, the block header is a fixed length structure. Each header consists of 208 bytes.

Many extensions require adding new fields into the block, but there’s no enough reserved bits for them. For example, flyclient requires a 64-byte hash in the header.
Workaround exists such as storing these data in the cellbase transaction, but it has a big overhead for clients which want to verify the chain using PoW only. Because they have to download the cellbase transaction and the merkle tree proof of the cellbase transaction, which can be larger than the block header itself.

This document proposes a solution to add a variable length field in the block. How to interpret the new field is beyond the scope of this document and must be defined and deployed via a future soft fork. Although the field is added to the block body, nodes can synchronize the block header and this field together in the future version.

## Specification

The block header is encoded as a molecule struct, which consists of fixed length fields. The header binary is just the concatenation of all the fields in sequence.

There are many ways to add the variable length field to the block header. This RFC proposes to replace the `uncles_hash` in the header with the new field `extra_hash`, which is also a 32-byte hash. The block will have a new field `extension`.

There are two important time points to deploy this RFC, activation epoch A and extension application epoch B.

In blocks before epoch A, the `extension` must be absent. The value of `extra_hash` is the same as the original `uncles_hash` in these blocks, so this RFC will not change the serialized headers of existing blocks. The field `extra_hash` is all zeros when the `uncles` field is empty, or `ckbhash` on all the uncle header hashes concatenated together.

```
uncles_hash = 0 when uncles is empty, otherwise

uncles_hash = ckbhash(U1 || U2 || ... || Un)
where Ui is the header_hash of the i-th uncle in uncles
```

See Appendix for the default hash function `ckbhash`. The annotation `||` means bytes concatenation.

In blocks generated since epoch A, `extension` can be absent, or any binary with 1 to 96 bytes. The upper limit 96 prevents abusing this field because there's no consensus rule to verify the content of `extension`. The 96 bytes limit allows storing the 64-byte flyclient hash and an extra 32-byte hash on further extension bytes.

The `extra_hash` is defined as:

* When `extension` is empty, `extra_hash` is the same as the `uncles_hash`.
* Otherwise `extra_hash = ckbhash(uncles_hash || ckbhash(extension))`

Since epoch B, consensus will define the schema of `extension` and verify the content. This is a soft fork if the `extension` is at most 96 bytes, because nodes deployed since epoch A do not verify the content of `extension`.

### P2P Protocols Changes

The field `uncles_hash` in the block header is renamed to `extra_hash`.

```
struct RawHeader {
version: Uint32,
compact_target: Uint32,
timestamp: Uint64,
number: Uint64,
epoch: Uint64,
parent_hash: Byte32,
transactions_root: Byte32,
proposals_hash: Byte32,
extra_hash: Byte32,
dao: Byte32,
}
```

The new field `extension` will be added to the block body and following data structures:

```
table Block {
header: Header,
uncles: UncleBlockVec,
transactions: TransactionVec,
proposals: ProposalShortIdVec,
extension: Bytes,
}

table CompactBlock {
header: Header,
short_ids: ProposalShortIdVec,
prefilled_transactions: IndexTransactionVec,
uncles: Byte32Vec,
proposals: ProposalShortIdVec,
extension: Bytes,
}
```

For blocks before the activation epoch A, `extension` must be absent. After activation, the node must verify that `extension` is absent or a binary with 1 to 96 bytes, and `uncles` and `extension` match the `extra_hash` in the header.

Pay attention that the `extension` field will occupy the block size. See section [Block and Compact Block Structure](../0020-ckb-consensus-protocol/0020-ckb-consensus-protocol.md#block-and-compact-block-structure) in RFC20 for details.

The uncle blocks packaged in `uncles` will not include the `extension` field.

### RPC Changes

* The `uncles_hash` is renamed to `extra_hash`.
* The new field `extension` is added to the block body RPC response. For blocks generated in ckb2019, it is always empty.

## Comparison With Alternative Solutions

1. [Appending the Field At the End](./1-appending-the-field-at-the-end.md)
2. [Using Molecule Table in New Block Headers](./2-using-molecule-table-in-new-block-headers.md)
3. [Appending a Hash At the End](./3-appending-a-hash-at-the-end.md)

## Test Vectors

### Block Hash

<details><summary>Block Template</summary>

```json
{
"version": "0x0",
"compact_target": "0x20010000",
"current_time": "0x17af3f66555",
"number": "0x3",
"epoch": "0x3e80003000000",
"parent_hash": "0xebf229020f333100942279dc33303ae0dfcbe720d8d11818687e6654c157294c",
"cycles_limit": "0x2540be400",
"bytes_limit": "0x91c08",
"uncles_count_limit": "0x2",
"uncles": [],
"transactions": [
{
"hash": "0x9110ca9266f89938f09ae6f93cc914b2c856cc842440d56fda6d16ee62543f5c",
"required": false,
"cycles": "0x19f2d1",
"depends": null,
"data": {
"version": "0x0",
"cell_deps": [
{
"out_point": {
"tx_hash": "0xace5ea83c478bb866edf122ff862085789158f5cbff155b7bb5f13058555b708",
"index": "0x0"
},
"dep_type": "dep_group"
}
],
"header_deps": [],
"inputs": [
{
"since": "0x0",
"previous_output": {
"tx_hash": "0xa563884b3686078ec7e7677a5f86449b15cf2693f3c1241766c6996f206cc541",
"index": "0x7"
}
}
],
"outputs": [
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649",
"hash_type": "data",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
},
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8",
"hash_type": "type",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
},
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649",
"hash_type": "data1",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
}
],
"outputs_data": [
"0x",
"0x",
"0x"
],
"witnesses": [
"0x550000001000000055000000550000004100000070b823564f7d1f814cc135ddd56fd8e8931b3a7040eaf1fb828adae29736a3cb0bc7f65021135b293d10a22da61fcc64f7cb660bf2c3276ad63630dad0b6099001"
]
}
}
],
"proposals": [],
"cellbase": {
"hash": "0x185d1c46fe3c4a0a1a5ae47203df2aeebbb97ac353abcf2c6a3fc2548ecd4eda",
"cycles": null,
"data": {
"version": "0x0",
"cell_deps": [],
"header_deps": [],
"inputs": [
{
"since": "0x3",
"previous_output": {
"tx_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"index": "0xffffffff"
}
}
],
"outputs": [],
"outputs_data": [],
"witnesses": [
"0x590000000c00000055000000490000001000000030000000310000009bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce80114000000c8328aabcd9b9e8e64fbc566c4385c3bdeb219d700000000"
]
}
},
"work_id": "0x2",
"dao": "0x105cabf31c1fa12eacfa6990f2862300bdaf44b932000000008d5fff03fbfe06",
"extension": "0x626c6f636b202333"
}
```

</details>

<details><summary>Block</summary>

```json
{
"header": {
"version": "0x0",
"compact_target": "0x20010000",
"timestamp": "0x17af3f66555",
"number": "0x3",
"epoch": "0x3e80003000000",
"parent_hash": "0xebf229020f333100942279dc33303ae0dfcbe720d8d11818687e6654c157294c",
"transactions_root": "0x0bbf9d8946932c9c33a46c8d13b9ecfcf850ccc1728fc9c9c5d14710ad9428ad",
"proposals_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"extra_hash": "0xfbbfbaaa0afac7730f4a6102b376986f1f288f3eccb18e0d16d58422aab28aad",
"dao": "0x105cabf31c1fa12eacfa6990f2862300bdaf44b932000000008d5fff03fbfe06",
"nonce": "0x6e43a02f3ed8bb00dea7f78c12fe94f5"
},
"uncles": [],
"transactions": [
{
"version": "0x0",
"cell_deps": [],
"header_deps": [],
"inputs": [
{
"since": "0x3",
"previous_output": {
"tx_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"index": "0xffffffff"
}
}
],
"outputs": [],
"outputs_data": [],
"witnesses": [
"0x590000000c00000055000000490000001000000030000000310000009bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce80114000000c8328aabcd9b9e8e64fbc566c4385c3bdeb219d700000000"
]
},
{
"version": "0x0",
"cell_deps": [
{
"out_point": {
"tx_hash": "0xace5ea83c478bb866edf122ff862085789158f5cbff155b7bb5f13058555b708",
"index": "0x0"
},
"dep_type": "dep_group"
}
],
"header_deps": [],
"inputs": [
{
"since": "0x0",
"previous_output": {
"tx_hash": "0xa563884b3686078ec7e7677a5f86449b15cf2693f3c1241766c6996f206cc541",
"index": "0x7"
}
}
],
"outputs": [
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649",
"hash_type": "data",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
},
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8",
"hash_type": "type",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
},
{
"capacity": "0x2540be400",
"lock": {
"code_hash": "0x709f3fda12f561cfacf92273c57a98fede188a3f1a59b1f888d113f9cce08649",
"hash_type": "data1",
"args": "0xc8328aabcd9b9e8e64fbc566c4385c3bdeb219d7"
},
"type": null
}
],
"outputs_data": [
"0x",
"0x",
"0x"
],
"witnesses": [
"0x550000001000000055000000550000004100000070b823564f7d1f814cc135ddd56fd8e8931b3a7040eaf1fb828adae29736a3cb0bc7f65021135b293d10a22da61fcc64f7cb660bf2c3276ad63630dad0b6099001"
]
}
],
"proposals": [],
"extension": "0x626c6f636b202333"
}
```

</details>

The hashes:

```
Block Hash:
0xb93dad02d24e9d30c49023d08f84dd8ec34118c1bfec9ed432b75619964686c3

Transaction Hashes:
0x185d1c46fe3c4a0a1a5ae47203df2aeebbb97ac353abcf2c6a3fc2548ecd4eda
0x9110ca9266f89938f09ae6f93cc914b2c856cc842440d56fda6d16ee62543f5c
```

## Appendix

### ckbhash
doitian marked this conversation as resolved.
Show resolved Hide resolved

CKB uses [blake2b](https://blake2.net/blake2.pdf) as the default hash algorithm with following configurations:

- output digest size: 32
- personalization: ckb-default-hash

Python 3 Example and test vectors:

```python
import hashlib
import unittest

def ckbhash():
return hashlib.blake2b(digest_size=32, person=b'ckb-default-hash')

class TestCKBBlake2b(unittest.TestCase):

def test_empty_message(self):
hasher = ckbhash()
hasher.update(b'')
self.assertEqual('44f4c69744d5f8c55d642062949dcae49bc4e7ef43d388c5a12f42b5633d163e', hasher.hexdigest())

if __name__ == '__main__':
unittest.main()
```
Loading