[WIP] Hybrid Casper FFG Implementation #8654

sorpaas · 2018-05-18T02:40:54Z

cc #7162, djrtwo/EIPs#5. This is a work-in-progress. Do not merge. Reviews/comments/grumbles are highly appreciated.

This implements EIP-1011. Currently we have initialization/deployment, vote transactions validation/sorting, fork choice and finalization implemented. Command line options are not added and there's no block reward issuance modifications for ease of testing. EIP-1011's current specification is not compatible with the old testnet, so only local testing is possible.

Current known issues:

Snapshot, warp sync, light client won't work with Casper right now.
Miner's vote transaction selection is really inefficient. It probably only works for local vote transactions now. We might need a new struct TransactionPicker which picks the next transaction to be included in a block.
Miner's block reopening won't work with Casper. Because of the introduction of vote transaction ordering, when a new transaction comes, we cannot just reopen the current sealing block, revert to the last transaction's executed state, execute the new transactions and then close the block. Whenever we need to do this, it seems that we need to re-execute at least all of the vote transactions. @djrtwo do you have any suggestions how this might be fixed?

The current testnet seems to be outdated. So currently we use a tmp spec for testing.

As many comments noted at vyperlang/vyper#566, those two are mostly equivalent except edge cases. So it's good enough for testing.

ethabi does not support the later yet. We just need to remember that it has 10 decimal places.

Otherwise it looks like broken

5chdn · 2018-05-18T06:07:16Z

EIP-1011's current specification is not compatible with the old testnet, so only local testing is possible.

Well, let's start an EIP-1011 testnet then? Happy to set this up. Other clients working on that spec can join at any point.

sorpaas · 2018-05-18T07:03:34Z

@5chdn Need some more polishing and testing, and in particular, we need to have a validator implementation (probably just an external one through JSONRPC endpoints). Otherwise we cannot interact with the Casper contract. Many of the current parameters used here will also likely need to be changed because EIP1011 has not been finalized. But anyway, no matter testing locally, privately or publicly, I think having a testnet is important, and I'll definitely try to get it done soon.

Besides, I think @djrtwo, and in particular, the py-evm group is planning to launch another new testnet soon. So we may also consider to join that.

djrtwo · 2018-05-18T07:53:54Z

@5chdn Yeah, the testnet will be pretty boring without validators 😝. There was one written in pyethapp that is now deprecated. Harmony was also implementing one, but it was for the pre-EIP spec.

Vyper just released some updates that we needed on the python side of things to get the EIP implementation going in py-evm. Trying to get out any implementation next week and start coordinating a multi-client testnet from there.

@sorpaas You should theoretically be able to open up the block, add a nonvote tx and not re-run the votes. Casper vote txs can affect block transaction execution, but normal block transactions cannot affect casper votes. Essentially the post block state is all the normal transactions + limited state modification from the votes (state that the normal txs can't touch). The limited state modification from the votes will be the same regardless of what normal transactions were placed in front of it. I'm not sure how the internals of parity work, but you should be able apply with vote state mod to any-prestate without re-running.

sorpaas · 2018-05-29T08:22:36Z

A note on transaction queue: currently we mark vote transactions as service transactions, so propagation should have no issue, and we can implement DoS methods with this. However, to "fully fill" a block with vote transactions, this may require to extend MAX_SKIPPED_TRANSACTIONS to a much larger value. So eventually this needs to be replaced by two transaction queues.

We can use json config to customize them now.

sorpaas · 2018-05-30T06:33:38Z

cc @5chdn

Just spinned up a testnet using this branch so that we can test out whether the networking and txpool propagation works alright, before we get the testnet from @djrtwo. 😃 You can also use this to test any validator implementation against Casper revision 0.2.0. Please PM me if you need any testnet tokens. :)

Network stats: http://139.162.71.88:3000/
Validator log: https://seashells.io/v/Qbcqk3EV

Specifications:

Use this file and binary from this branch casper to connect to the network.
The SimpleCasper contract is compiled using master Vyper with default settings at tag v0.2.0, and is at address 0x40.
The PurityChecker contract is at address 0x41.
The MsgHasher contract is at address 0x42.
The RlpDecoder contract is at address 0x43.
Casper transition happened at block 50, and the warm up period is 50.
Other parameters are the same as in EIP1011, except the vote transaction ordering requirement is waived, which according to @djrtwo, will be reflected in a future version of the EIP1011 spec.
Block reward reduction is not handled for easy testing.
EIP155 is 0x1ee7.

…sper

afck · 2018-06-12T09:54:08Z

ethcore/src/engines/hybrid_casper.rs

+
+	/// Casper-specific fork choice.
+	pub fn fork_choice(&self, new: &ExtendedHeader, current: &ExtendedHeader) -> ForkChoice {
+		let new_metadata: HybridCasperMetadata = new.metadata().map(|d| rlp::decode(d).expect("Metadata is only set by serializing CasperMetadata struct; deserailzling CasperMetadata RLP always succeeds; qed")).unwrap_or(Default::default());


Typo: "deserailzling" (also several times below)

afck · 2018-06-12T09:58:13Z

ethcore/src/engines/hybrid_casper.rs

+
+		let mut metadata: HybridCasperMetadata = block.metadata().map(|d| rlp::decode(d).expect("Metadata is only set by serializing CasperMetadata struct; deserailzling CasperMetadata RLP always succeeds; qed")).unwrap_or(Default::default());
+		metadata.vote_gas_used = receipt.gas_used;
+		receipt.gas_used = block.receipts().last().map(|r| r.gas_used).unwrap_or(U256::zero());


Maybe unwrap_or_else(U256::zero) would be better: It avoids creating the zero U256 if there was a receipt.

afck · 2018-06-12T10:50:55Z

ethcore/src/engines/hybrid_casper.rs

+
+	/// Casper-specific fork choice.
+	pub fn fork_choice(&self, new: &ExtendedHeader, current: &ExtendedHeader) -> ForkChoice {
+		let new_metadata: HybridCasperMetadata = new.metadata().map(|d| rlp::decode(d).expect("Metadata is only set by serializing CasperMetadata struct; deserailzling CasperMetadata RLP always succeeds; qed")).unwrap_or(Default::default());


unwrap_or_else(Default::default) would avoid unnecessarily instantiating HybridCasperMetadata (also a few times below).

…sper

5chdn · 2018-06-18T13:51:39Z

Putting this on ice as per dev call last week. Are you continuing work on this branch or shall we close it?

sorpaas · 2018-06-18T15:23:22Z

No. Let's close this for now.
I think the beacon chain design would require a completely different set of code.

5chdn · 2018-06-18T18:49:51Z

😢

sorpaas added 24 commits May 17, 2018 15:44

Create a basic spec for casper testing

209ed4a

The current testnet seems to be outdated. So currently we use a tmp spec for testing.

Add basic hybridCasperTransition flag

dc1a3de

Refactor contract hex code and update casper ones

b8c6dc5

Fix missing imports

965f08f

Init casper contract code at the transition block

5df4948

Init casper contract abi

0610c8f

Use fixed168x10 to replace decimal10 temporarily

a6a63aa

As many comments noted at vyperlang/vyper#566, those two are mostly equivalent except edge cases. So it's good enough for testing.

Use double quotes to make it valid json

4e56fd7

Use int168 to replace fixed168x10

29cd6f8

ethabi does not support the later yet. We just need to remember that it has 10 decimal places.

Call init function in Casper

8b283b6

Lower initial difficulty for casper local testnet

6343894

Increase the difficulty a little

86e6377

Otherwise it looks like broken

Tweaking the testnet config

77f1275

Add basic framework for transaction reordering

4eeb6d5

Handle transaction ordering in miner

a7a02ca

Verify transaction ordering in verifier

3bf06f6

Delegate all work to EthereumMachine

96ad2b5

Implement vote_gas_used through metadata

d3d5b5d

Implement vote transaction check

39f51f4

Implement all vote transaction related features

5b6f904

Set casper contract balance

d58e5ca

Call init_epoch for epoch length

837612b

Implement epoch score fork choice

8f509be

Handle Casper finalization

fc47b49

sorpaas added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. M4-core ⛓ Core client code / Rust. labels May 18, 2018

5chdn added this to the 1.12 milestone May 18, 2018

sorpaas added 5 commits May 29, 2018 08:49

Fix all missing docs

58875db

Fix tests

bfd1b0c

Use saturating_add to avoid possible overflow

1319068

Allow engine to specify additional transactions as service trnsaction

e67ff23

Mark casper vote transactions as service transaction

3f41dbc

sorpaas added 2 commits May 30, 2018 09:10

Set casper values back to defaults

e10bb64

We can use json config to customize them now.

Update testnet definition

eaadeee

This comment has been minimized.

Sign in to view

sorpaas added 5 commits June 4, 2018 11:31

Merge branch 'master' of https://github.com/paritytech/parity into ca…

4ce510e

…sper

Fix cargo.lock merge failure

5c7d64e

Fix casper test config

1ba5e4d

Merge branch 'master' of https://github.com/paritytech/parity into ca…

55f4e3b

…sper

Fix tests in EIP86 incident

801a964

afck reviewed Jun 12, 2018

View reviewed changes

sorpaas added 4 commits June 12, 2018 19:54

Fix all instance of typo for "deserializing"

cfe51a8

Use unwrap_or_else whenever possible

7bfd990

Update casper_test file to point to the current live testnet

5097bc6

Merge branch 'master' of https://github.com/paritytech/parity into ca…

773ba57

…sper

mkalinin mentioned this pull request Jun 14, 2018

Casper: Parity testnet integation work ethereum/ethereumj#1097

Closed

5chdn added A1-onice 🌨 Pull request is reviewed well, but should not yet be merged. and removed A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. labels Jun 18, 2018

sorpaas closed this Jun 18, 2018

afck referenced this pull request in poanetwork/hbbft Jun 25, 2018

Added structured logging for reporting faulty nodes.

072d470

sorpaas mentioned this pull request Jul 19, 2018

block cleanup #9117

Merged

5chdn deleted the casper branch November 1, 2018 10:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Hybrid Casper FFG Implementation #8654

[WIP] Hybrid Casper FFG Implementation #8654

sorpaas commented May 18, 2018 •

edited by rphmeier

Loading

5chdn commented May 18, 2018

sorpaas commented May 18, 2018 •

edited

Loading

djrtwo commented May 18, 2018 •

edited

Loading

sorpaas commented May 29, 2018

sorpaas commented May 30, 2018 •

edited

Loading

This comment has been minimized.

afck Jun 12, 2018

afck Jun 12, 2018

afck Jun 12, 2018

5chdn commented Jun 18, 2018

sorpaas commented Jun 18, 2018

5chdn commented Jun 18, 2018

[WIP] Hybrid Casper FFG Implementation #8654

[WIP] Hybrid Casper FFG Implementation #8654

Conversation

sorpaas commented May 18, 2018 • edited by rphmeier Loading

5chdn commented May 18, 2018

sorpaas commented May 18, 2018 • edited Loading

djrtwo commented May 18, 2018 • edited Loading

sorpaas commented May 29, 2018

sorpaas commented May 30, 2018 • edited Loading

This comment has been minimized.

afck Jun 12, 2018

Choose a reason for hiding this comment

afck Jun 12, 2018

Choose a reason for hiding this comment

afck Jun 12, 2018

Choose a reason for hiding this comment

5chdn commented Jun 18, 2018

sorpaas commented Jun 18, 2018

5chdn commented Jun 18, 2018

sorpaas commented May 18, 2018 •

edited by rphmeier

Loading

sorpaas commented May 18, 2018 •

edited

Loading

djrtwo commented May 18, 2018 •

edited

Loading

sorpaas commented May 30, 2018 •

edited

Loading