Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSZ implementation for exec. spec - Support for Python 3 typing. #1077

Merged
merged 52 commits into from
Jun 4, 2019
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
761c9e5
SSZ impl. rework started, see issue 1064
protolambda May 12, 2019
08faa86
POC ssz types spec build + update spec defs, typing still needs work
protolambda May 12, 2019
ed4416b
update SSZ implementation
protolambda May 24, 2019
81cb4a2
update some common usage of SSZ types, as a start
protolambda May 24, 2019
5b6a98b
Some updates
vbuterin May 25, 2019
3b4d9b7
Class-ified the type definitions
vbuterin May 25, 2019
e14f789
Presentation edits
vbuterin May 27, 2019
d5eab25
improve impl, box less used integer types, use uint64 as default
protolambda May 27, 2019
f308888
Minor adjustments + getting the beacon spec doc ready
protolambda May 27, 2019
132d3c9
fix spec builder
protolambda May 27, 2019
d63b553
efficiency bugfix in bytes encoding, improve typing doc + bugfix
protolambda May 27, 2019
87b3466
update encoder and decoder for reading from parsed data
protolambda May 27, 2019
c68944b
separate type (direct) from kinds (alias incl)
protolambda May 27, 2019
0f79ed7
update yaml encoder/decoder and obj randomizer for typed ssz usage
protolambda May 27, 2019
d023d2d
lots of bugfixes
protolambda May 27, 2019
b4c4df6
bugfix in sss_types global building in script
protolambda May 27, 2019
54b14b5
update ssz-static generator
protolambda May 27, 2019
0e9a047
Merge branch 'dev' into ssz-impl-rework
protolambda May 27, 2019
9e61cc2
bugfix: typing edge-case: python bool is subclass of int.
protolambda May 27, 2019
5e28adf
bugfix: don't forget about var-length bytes getting a length mixin
protolambda May 27, 2019
00ffaf4
Fix ssz path
hwwhww May 28, 2019
f0ceefc
Make `test_libs/pyspec/tests` pass
hwwhww May 28, 2019
19601df
Starting work on partials
vbuterin May 28, 2019
4955c40
merge
vbuterin May 28, 2019
053d6e7
Simplified hash tree root
vbuterin May 28, 2019
ebdf74c
kick the CI
hwwhww May 30, 2019
8fae0f8
Added support for SSZ partials
vbuterin May 30, 2019
3630b2e
Merge branch 'ssz-impl-rework' of github.com:ethereum/eth2.0-specs in…
vbuterin May 30, 2019
e044305
Merge branch 'dev' into ssz-impl-rework
protolambda May 31, 2019
3cb43fc
fix ssz infer type decorator, one None argument too many
protolambda May 31, 2019
1c734d3
fix utils init
protolambda May 31, 2019
33233c9
fix ssz imports
protolambda May 31, 2019
77d4f0b
update generator type hinting to use new type syntax
protolambda May 31, 2019
df25f22
get rid of scope-problem inducing init trick
protolambda May 31, 2019
7df788c
resolve linting problems, except ssz-partials
protolambda Jun 1, 2019
f3a517b
fix minor lint problems
protolambda Jun 1, 2019
b79f01e
update flake8 to support type annotation in linting, ignore operator …
protolambda Jun 1, 2019
fae1e92
resolve some new lint issues detected by newer version, and a few loo…
protolambda Jun 1, 2019
8e8ef2d
Fix forgotten setup.py change, make generators using ssz work again
protolambda Jun 1, 2019
a557605
Fix `ssz_partials.py` linter errors
hwwhww Jun 3, 2019
6cd9811
Delete extra newline
hwwhww Jun 3, 2019
fb584bc
fix linting dependency + caching issue
protolambda Jun 3, 2019
a7ee6f1
Refactor
hwwhww Jun 4, 2019
e2eab66
Refactor
hwwhww Jun 4, 2019
6d55ba9
minor refactor
hwwhww Jun 4, 2019
8631cad
Apply suggestions from code review
protolambda Jun 4, 2019
fe9c708
Fix whitespace
protolambda Jun 4, 2019
59137fd
fix get_zero_value exception raise + fix up type checks
protolambda Jun 4, 2019
578328b
call comments: remove partials from ssz rework PR, review that sepera…
protolambda Jun 4, 2019
6168a90
speed and simplicity improvement for next_power_of_two function
protolambda Jun 4, 2019
4bf3a26
fix formatting
protolambda Jun 4, 2019
9bdb182
remove tautological type definition
protolambda Jun 4, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions scripts/phase0/build_spec.py
Original file line number Diff line number Diff line change
@@ -33,9 +33,9 @@ def build_phase0_spec(sourcefile, outfile):
Shard = NewType('Shard', int) # uint64
ValidatorIndex = NewType('ValidatorIndex', int) # uint64
Gwei = NewType('Gwei', int) # uint64
Bytes32 = NewType('Bytes32', bytes) # bytes32
BLSPubkey = NewType('BLSPubkey', bytes) # bytes48
BLSSignature = NewType('BLSSignature', bytes) # bytes96
Bytes32 = BytesN[32]
BLSPubkey = NewType('BLSPubkey', BytesN[48])
BLSSignature = NewType('BLSSignature', BytesN[96])
Store = None
""")

18 changes: 0 additions & 18 deletions scripts/phase0/function_puller.py
Original file line number Diff line number Diff line change
@@ -2,22 +2,6 @@
from typing import List


def translate_ssz_type_line(line: str) -> str:
if ':' not in line:
return line
start = line[:line.index(':')]
field_type = line[line.index(':')+2:]
if field_type.startswith('['):
if ',' in line:
# TODO: translate [Foobar, SOME_THING] to Vector[Foobar, SSZLen(SOME_THING)] cleanly.
# just brute it here
field_type = 'Vector[%s, SSLen(%s)]' % (field_type[1:field_type.index(',')], field_type[field_type.index(',')+2:len(field_type)-1])
else:
field_type = 'List[%s]' % field_type[1:len(field_type)-1]
line = start + ': ' + field_type
return line


def get_spec(file_name: str) -> List[str]:
code_lines = []
pulling_from = None
@@ -51,8 +35,6 @@ def get_spec(file_name: str) -> List[str]:
if line[:3] == 'def':
code_lines.append('')
code_lines.append('')
if current_typedef is not None:
line = translate_ssz_type_line(line)
code_lines.append(line)
elif pulling_from is None and len(line) > 0 and line[0] == '|':
row = line[1:].split('|')
48 changes: 24 additions & 24 deletions specs/core/0_beacon-chain.md
Original file line number Diff line number Diff line change
@@ -340,8 +340,8 @@ The types are defined topologically to aid in facilitating an executable version
```python
{
# Validator indices
custody_bit_0_indices: [uint64]
custody_bit_1_indices: [uint64]
custody_bit_0_indices: List[uint64]
custody_bit_1_indices: List[uint64]
# Attestation data
data: AttestationData
# Aggregate signature
@@ -418,9 +418,9 @@ The types are defined topologically to aid in facilitating an executable version
```python
{
# Block roots
block_roots: [bytes32, SLOTS_PER_HISTORICAL_ROOT]
block_roots: Vector[bytes32, SLOTS_PER_HISTORICAL_ROOT]
# State roots
state_roots: [bytes32, SLOTS_PER_HISTORICAL_ROOT]
state_roots: Vector[bytes32, SLOTS_PER_HISTORICAL_ROOT]
}
```

@@ -470,7 +470,7 @@ The types are defined topologically to aid in facilitating an executable version
```python
{
# Branch in the deposit tree
proof: [bytes32, DEPOSIT_CONTRACT_TREE_DEPTH]
proof: Vector[bytes32, DEPOSIT_CONTRACT_TREE_DEPTH]
# Index in the deposit tree
index: uint64
# Data
@@ -521,12 +521,12 @@ The types are defined topologically to aid in facilitating an executable version
randao_reveal: bytes96
eth1_data: Eth1Data
graffiti: bytes32
proposer_slashings: [ProposerSlashing]
attester_slashings: [AttesterSlashing]
attestations: [Attestation]
deposits: [Deposit]
voluntary_exits: [VoluntaryExit]
transfers: [Transfer]
proposer_slashings: List[ProposerSlashing]
attester_slashings: List[AttesterSlashing]
attestations: List[Attestation]
deposits: List[Deposit]
voluntary_exits: List[VoluntaryExit]
transfers: List[Transfer]
}
```

@@ -555,16 +555,16 @@ The types are defined topologically to aid in facilitating an executable version
fork: Fork # For versioning hard forks

protolambda marked this conversation as resolved.
Show resolved Hide resolved
# Validator registry
validator_registry: [Validator]
balances: [uint64]
validator_registry: List[Validator]
balances: List[uint64]

protolambda marked this conversation as resolved.
Show resolved Hide resolved
# Randomness and committees
latest_randao_mixes: [bytes32, LATEST_RANDAO_MIXES_LENGTH]
latest_randao_mixes: Vector[bytes32, LATEST_RANDAO_MIXES_LENGTH]
latest_start_shard: uint64

protolambda marked this conversation as resolved.
Show resolved Hide resolved
# Finality
previous_epoch_attestations: [PendingAttestation]
current_epoch_attestations: [PendingAttestation]
previous_epoch_attestations: List[PendingAttestation]
current_epoch_attestations: List[PendingAttestation]
previous_justified_epoch: uint64
current_justified_epoch: uint64
previous_justified_root: bytes32
@@ -574,20 +574,20 @@ The types are defined topologically to aid in facilitating an executable version
finalized_root: bytes32

protolambda marked this conversation as resolved.
Show resolved Hide resolved
# Recent state
current_crosslinks: [Crosslink, SHARD_COUNT]
previous_crosslinks: [Crosslink, SHARD_COUNT]
latest_block_roots: [bytes32, SLOTS_PER_HISTORICAL_ROOT]
latest_state_roots: [bytes32, SLOTS_PER_HISTORICAL_ROOT]
latest_active_index_roots: [bytes32, LATEST_ACTIVE_INDEX_ROOTS_LENGTH]
current_crosslinks: Vector[Crosslink, SHARD_COUNT]
previous_crosslinks: Vector[Crosslink, SHARD_COUNT]
latest_block_roots: Vector[bytes32, SLOTS_PER_HISTORICAL_ROOT]
latest_state_roots: Vector[bytes32, SLOTS_PER_HISTORICAL_ROOT]
latest_active_index_roots: Vector[bytes32, LATEST_ACTIVE_INDEX_ROOTS_LENGTH]
# Balances slashed at every withdrawal period
latest_slashed_balances: [uint64, LATEST_SLASHED_EXIT_LENGTH]
latest_slashed_balances: Vector[uint64, LATEST_SLASHED_EXIT_LENGTH]
# `latest_block_header.state_root == ZERO_HASH` temporarily
latest_block_header: BeaconBlockHeader
historical_roots: [bytes32]
historical_roots: List[bytes32]

protolambda marked this conversation as resolved.
Show resolved Hide resolved
# Ethereum 1.0 chain data
latest_eth1_data: Eth1Data
eth1_data_votes: [Eth1Data]
eth1_data_votes: List[Eth1Data]
deposit_index: uint64
}
```