decode block rlp less often #9252

debris · 2018-07-30T14:17:25Z

in total:

removed 4 redundant rlp deserializations
avoid 1 redundant block data copy

debris · 2018-07-30T14:19:03Z

ethcore/src/block.rs

-			let uncle_bytes = encode_list(&s.block.uncles);
-			s.block.header.set_uncles_hash(keccak(&uncle_bytes));
-		}
-		if s.block.header.receipts_root().is_zero() || s.block.header.receipts_root() == &KECCAK_NULL_RLP {


as mentioned before, this checks could lead to invalid block being locked, so I removed them

debris · 2018-07-30T14:19:37Z

ethcore/src/block.rs

@@ -537,18 +514,16 @@ impl LockedBlock {
 		self,
 		engine: &EthEngine,
 		seal: Vec<Bytes>,
-	) -> Result<SealedBlock, (Error, LockedBlock)> {


LockedBlock returned in Err was never used

debris · 2018-07-30T14:19:59Z

ethcore/src/block.rs


 	enact(
 		block.header,
 		block.transactions,
-		view.uncles(),
+		block.uncles,


removed redundant decoding of uncles

debris · 2018-07-30T14:21:12Z

ethcore/src/client/client.rs

-		use verification::queue::kind::blocks::Unverified;
-
-		// create unverified block here so the `keccak` calculation can be cached.
-		let unverified = Unverified::from_rlp(bytes)?;


removed redundant block deserialization

debris · 2018-07-30T14:21:26Z

ethcore/src/client/client.rs

 		trace_time!("queue_ancient_block");
-		let header: Header = ::rlp::Rlp::new(&block_bytes).val_at(0)?;


removed redundant header deserialization

debris · 2018-07-30T14:22:53Z

ethcore/src/tests/client.rs

@@ -107,24 +107,6 @@ fn imports_good_block() {
 	assert!(!block.into_inner().is_empty());
 }

-#[test]
-fn fails_to_import_block_with_invalid_rlp() {


this test is not needed, cause import_block takes already deserialized block as a param

debris · 2018-07-30T14:23:43Z

ethcore/sync/src/block_sync.rs

-			let (h, number, parent) = {
-				let header = view!(BlockView, &block).header_view();
-				(header.hash(), header.number(), header.parent_hash())
+			let block = match Unverified::from_rlp(block) {


Block::is_good was doing full block deserialization. It's now replaced with Unverified::from_rlp

debris · 2018-07-30T14:23:55Z

ethcore/sync/src/block_sync.rs

-			}
-
-			let (h, number, parent) = {
-				let header = view!(BlockView, &block).header_view();


removed redundant deserialization

debris · 2018-07-30T14:24:22Z

ethcore/sync/src/blocks.rs

@@ -294,7 +294,7 @@ impl BlockCollection {
 				let header = view!(HeaderView, &block.header);
 				let block_view = Block::new_from_header_and_body(&header, &body);
 				drained.push(BlockAndReceipts {
-					block: block_view.rlp().as_raw().to_vec(),
+					block: block_view.into_inner(),


move data instead of copying it

sorpaas

LGTM overall. A small grumble on close's additional clones.

sorpaas · 2018-07-31T05:32:59Z

ethcore/src/block.rs

-		}));
-		s.block.header.set_gas_used(s.block.receipts.last().map_or_else(U256::zero, |r| r.gas_used));
-
+		let closed = self.close()?;


Just want to note that this will have one additional clone of unclosed_state (https://github.com/paritytech/parity-ethereum/blob/a19faf01f347e7f485a1e1a82c47e7715e132dba/ethcore/src/block.rs#L397), which is immediately dropped. Not sure whether it's optimized away by compiler.

Or maybe we can do it the other way around:

pub fn close(self) -> Result<ClosedBlock, Error> { let unclosed_state = self.block.state.clone(); let locked = self.close_and_lock()?; Ok(ClosedBlock { block: locked.block, unclosed_state, }) }

tomusdrw

Looks good, we should make sure that:

We can still sync the mainnet and Kovan (I remember some issues when I touched the close/close_and_lock ifs last time)
We can avoid the early allocation on_peer_new_block

tomusdrw · 2018-08-01T12:12:17Z

ethcore/src/client/client.rs


 			// Commit results
 			let mut batch = DBTransaction::new();
-			chain.insert_unordered_block(&mut batch, block, receipts, None, false, true);
+			chain.insert_unordered_block(&mut batch, encoded::Block::new(unverified.bytes), receipts, None, false, true);


So where does the verification actually happen?

4 lines above, we verify only header for old blocks. No logic has been changed here

tomusdrw · 2018-08-01T12:17:46Z

ethcore/sync/src/chain/handler.rs

-		let header: BlockHeader = header_rlp.as_val()?;
-		if header.number() > sync.highest_block.unwrap_or(0) {
-			sync.highest_block = Some(header.number());
+		let block = Unverified::from_rlp(r.at(0)?.as_raw().to_vec())?;


Can we avoid allocating here? We don't even know if the block is valid at this point.

good catch!

of course we can, I will fix it in the next pr as the allocation is not new (previously it happened in https://github.com/paritytech/parity-ethereum/pull/9252/files#diff-e192cabaad094a396a522f8a2633daceL184)

Yes, the point is that it was happening after the initial checks - currently we eagerly allocate for every incoming block.

but these initial checks check only block header, so if someone crafted a malicious block it would cause an allocation anyway

debris · 2018-08-01T13:49:29Z

just checked syncing Kovan. It looks solid,

./target/release/parity --no-warp --chain=kovan
2018-08-01 15:46:51  Starting Parity-Ethereum/v2.1.0-unstable-32f7b38-20180731/x86_64-macos/rustc1.27.0
2018-08-01 15:46:51  Keys path /Users/marek/Library/Application Support/io.parity.ethereum/keys/Kovan
2018-08-01 15:46:51  DB path /Users/marek/Library/Application Support/io.parity.ethereum/chains/kovan/db/9bf388941c25ea98
2018-08-01 15:46:51  State DB configuration: fast
2018-08-01 15:46:51  Operating mode: active
2018-08-01 15:46:51  Configured for Kovan using AuthorityRound engine
2018-08-01 15:46:56  Public node URL: enode://ba2dc102571706c562f1bcdc3cf73c8d22209a0b5b68455f74a54cd5463f65f75bc4a49a4e7f2dcb22b4f012f66b1e184ab7321ec54a5306c03c18e96625b61d@192.168.1.84:30303
2018-08-01 15:47:01  Syncing   #22757 0x3b4d…0f5f  2282.45 blk/s   39.7 tx/s    6 Mgas/s      0+ 5238 Qed    #27998    5/25 peers   6 MiB chain 51 MiB db 8 MiB queue 3 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:11  Syncing   #46910 0x3ba6…8b51  2418.69 blk/s  983.3 tx/s   34 Mgas/s      0+14686 Qed    #61725    5/25 peers   4 MiB chain 98 MiB db 22 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:21  Syncing   #73218 0x16d9…8b8a  2640.20 blk/s   15.1 tx/s    8 Mgas/s      0+18985 Qed    #92205    5/25 peers   4 MiB chain 97 MiB db 30 MiB queue 6 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:31  Syncing   #96512 0xcbaa…905d  2337.21 blk/s   56.0 tx/s   23 Mgas/s      0+22972 Qed   #119513    5/25 peers   4 MiB chain 94 MiB db 38 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:41  Syncing  #117422 0x4b13…2a94  2097.81 blk/s  118.5 tx/s   51 Mgas/s      0+23041 Qed   #140465    5/25 peers   3 MiB chain 93 MiB db 40 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:46  Syncing  #122637 0x45c6…76f2  1031.45 blk/s  233.6 tx/s   81 Mgas/s      0+22270 Qed   #144910    5/25 peers   4 MiB chain 93 MiB db 39 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:47:56  Syncing  #139557 0x8777…f784  1718.64 blk/s   70.4 tx/s   37 Mgas/s      0+21606 Qed   #161166    5/25 peers   3 MiB chain 93 MiB db 38 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:48:06  Syncing  #155433 0xd16c…69b4  1589.51 blk/s  241.1 tx/s   55 Mgas/s      0+21747 Qed   #177183    5/25 peers   3 MiB chain 93 MiB db 36 MiB queue 6 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:48:16  Syncing  #165918 0x8909…cbf8  1050.50 blk/s  116.1 tx/s   38 Mgas/s      0+25091 Qed   #191011    5/25 peers   4 MiB chain 94 MiB db 40 MiB queue 5 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:48:26  Syncing  #184425 0xe3b3…46ba  1854.78 blk/s  120.8 tx/s   29 Mgas/s      0+18527 Qed   #202955    5/25 peers   3 MiB chain 94 MiB db 32 MiB queue 8 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:48:36  Syncing  #198409 0x1a58…fd2c  1405.57 blk/s   97.7 tx/s   32 Mgas/s      0+ 4543 Qed   #202955    5/25 peers   4 MiB chain 94 MiB db 8 MiB queue 25 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
2018-08-01 15:48:46  Syncing  #213621 0xbcee…0a69  1524.10 blk/s  168.9 tx/s   65 Mgas/s      0+22601 Qed   #236225    5/25 peers   2 MiB chain 95 MiB db 40 MiB queue 8 MiB sync  RPC:  0 conn,    0 req/s,    0 µs
^C2018-08-01 15:48:51  Finishing work, please wait...

dvdplm

lgtm

dvdplm · 2018-08-02T08:06:35Z

ethcore/src/block.rs

-
-		let unclosed_state = s.block.state.clone();
-
-		s.engine.on_close_block(&mut s.block)?;


Where does this callback happen after this change?

https://github.com/paritytech/parity-ethereum/pull/9252/files#diff-e2e171033e005c3b329818aaff213767R408

dvdplm · 2018-08-02T08:12:49Z

ethcore/src/engines/validator_set/multi.rs

@@ -198,7 +199,7 @@ mod tests {
 		let sync_client = generate_dummy_client_with_spec_and_data(Spec::new_validator_multi, 0, 0, &[]);
 		sync_client.engine().register_client(Arc::downgrade(&sync_client) as _);
 		for i in 1..4 {
-			sync_client.import_block(client.block(BlockId::Number(i)).unwrap().into_inner()).unwrap();
+			sync_client.import_block(Unverified::from_rlp(client.block(BlockId::Number(i)).unwrap().into_inner()).unwrap()).unwrap();


This is a very long and noisy line...

dvdplm · 2018-08-02T08:17:21Z

ethcore/sync/src/block_sync.rs

+				Ok(block) => block,
+				Err(_) => {
+					debug!(target: "sync", "Bad block rlp");
+					bad = true;


This bad/ if bad {…} smells a little, especially in such a big function. How about just returning here and below with Err(BlockDownloaderImportError::Invalid);?

@dvdplm we still need to update the number of imported blocks before returning.

andresilva

LGTM

andresilva · 2018-08-02T09:17:47Z

ethcore/sync/src/block_sync.rs

+				Ok(block) => block,
+				Err(_) => {
+					debug!(target: "sync", "Bad block rlp");
+					bad = true;


@dvdplm we still need to update the number of imported blocks before returning.

…rp_sync_on_light_client * 'master' of https://github.com/paritytech/parity: Propagate transactions for next 4 blocks. (openethereum#9265) decode block rlp less often (openethereum#9252) Fix eternalities tests can_create (missing parameter) (openethereum#9270) Update ref to `parity-common` and update `seek` behaviour (openethereum#9257) Comply EIP-86 with the new definition (openethereum#9140) Check if synced when using eth_getWork (openethereum#9193) (openethereum#9210)

debris added 4 commits July 18, 2018 10:48

deserialize block only once during verification

eb78076

ethcore-sync uses Unverified

91edc42

Merge branch 'master' into use_unverified

0efdf0f

ethcore-sync uses Unverified

8290fb7

debris added A0-pleasereview 🤓 Pull request needs code review. M4-core ⛓ Core client code / Rust. labels Jul 30, 2018

debris changed the title ~~Use unverified~~ decode block rlp less often Jul 30, 2018

debris commented Jul 30, 2018

View reviewed changes

fixed build error

35fcb81

debris requested review from andresilva and sorpaas July 30, 2018 14:42

removed Block::is_good

a19faf0

sorpaas approved these changes Jul 31, 2018

View reviewed changes

5chdn added this to the 2.1 milestone Jul 31, 2018

applied review suggestions

32f7b38

debris added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jul 31, 2018

debris mentioned this pull request Aug 1, 2018

ethcore sync decodes rlp less often #9264

Merged

tomusdrw approved these changes Aug 1, 2018

View reviewed changes

dvdplm approved these changes Aug 2, 2018

View reviewed changes

andresilva approved these changes Aug 2, 2018

View reviewed changes

andresilva merged commit b4ae1b6 into master Aug 2, 2018

tomusdrw deleted the use_unverified branch August 2, 2018 09:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

decode block rlp less often #9252

decode block rlp less often #9252

debris commented Jul 30, 2018 •

edited

Loading

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

debris Jul 30, 2018

sorpaas left a comment

sorpaas Jul 31, 2018

sorpaas Jul 31, 2018

tomusdrw left a comment

tomusdrw Aug 1, 2018

debris Aug 1, 2018

tomusdrw Aug 1, 2018

debris Aug 1, 2018

tomusdrw Aug 1, 2018

debris Aug 1, 2018

debris commented Aug 1, 2018

dvdplm left a comment

dvdplm Aug 2, 2018

andresilva Aug 2, 2018

dvdplm Aug 2, 2018

dvdplm Aug 2, 2018

andresilva Aug 2, 2018

andresilva left a comment

andresilva Aug 2, 2018

		trace_time!("queue_ancient_block");
		let header: Header = ::rlp::Rlp::new(&block_bytes).val_at(0)?;


		let unclosed_state = s.block.state.clone();

		s.engine.on_close_block(&mut s.block)?;

decode block rlp less often #9252

decode block rlp less often #9252

Conversation

debris commented Jul 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorpaas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomusdrw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debris commented Aug 1, 2018

dvdplm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresilva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debris commented Jul 30, 2018 •

edited

Loading