messages from roothash mux app #2147

pro-wh · 2019-09-20T23:27:43Z

This adds a "roothash messages" field to the compute pipeline. A runtime would be able to send them, and they propagate into a block header.

Roothash messages are stored inline in the block header, rather than in storage.

This implements the simple merging algorithm, where up to one compute committee can send roothash messages.

This adds the staking general balance adjustment message, which causes the staking mux app to adjust an account's general balance.

There's a dummy acceptable transfer peer policy to control what runtimes can send these staking general balance adjustment messages. It accesses a set of approved runtime IDs in the genesis document.

TODO:

switch dummy acceptable transfer peer policy to deny all
remove dummy roothash message types
test unacceptable transfer peer
come up with a way to test without hacking up successfulRound

blocks #2077
context https://github.com/oasislabs/private-rfcs/pull/144

fixes #2230
fixes #2231

kostko · 2019-10-08T07:00:35Z

Roothash messages are stored inline in the block header, rather than in storage.

Would it make more sense for the messages to be in the Block rather than the Header?

kostko

Who will get slashed for reverting a round due to roothash messages not being accepted?

go/roothash/api/block/header.go

go/tendermint/apps/roothash/roothash.go

pro-wh · 2019-10-08T15:17:13Z

Who will get slashed for reverting a round due to roothash messages not being accepted?

maybe transaction scheduler, if we would add a rule that it must know that the sent roothash messages will be satisfactory.

pro-wh · 2019-10-08T15:18:23Z

Roothash messages are stored inline in the block header, rather than in storage.

Would it make more sense for the messages to be in the Block rather than the Header?

dunno. I can't tell why there's even a separation between block and header

kostko · 2019-10-08T15:35:25Z

dunno. I can't tell why there's even a separation between block and header

Yeah that's a good point, so previously bulk data was in the block while the header only included hashes. I guess this doesn't really make sense and maybe we should just remove one of the two to make stuff simpler.

From this standpoint just leave it in the header and maybe open an issue to remove/combine.

kostko · 2019-10-08T18:33:19Z

I've opened #2241.

pro-wh · 2019-10-09T00:31:29Z

Updates: now there's a prototype of loading an acceptable transfer peer policy from the genesis file. This adds a field that's a whitelist of runtime IDs.

go/roothash/api/block/header.go

codecov · 2019-10-09T22:21:04Z

Codecov Report

Merging #2147 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2147   +/-   ##
=======================================
  Coverage   53.79%   53.79%           
=======================================
  Files         263      263           
  Lines       28049    28049           
=======================================
  Hits        15088    15088           
  Misses      11430    11430           
  Partials     1531     1531

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dd842d9...546351e. Read the comment docs.

kostko

It would be useful to have an end-to-end test for roothash messages, so that the whole pipeline is tested.

pro-wh · 2019-10-16T00:13:07Z

added an e2e test. please review

kostko

Just minor nits.

go/oasis-node/cmd/debug/byzantine/compute.go

go/oasis-test-runner/scenario/e2e/roothash_messages.go

go/staking/api/api.go

go/tendermint/apps/staking/state.go

kostko · 2019-10-16T08:32:10Z

go/tendermint/apps/staking/state.go

+	// acceptableTransferPeersKeyFmt is the key format used for the acceptable transfer peers set.
+	//
+	// Value is a CBOR-serialized map from acceptable runtime IDs the boolean true.
+	acceptableTransferPeersKeyFmt = keyformat.New(0x55)


I assume the size of this map is expected to be small (on the order of number of runtimes)? Another option would be to have the entries be stored in the tree -- e.g., one key per entry:

keyformat.New(0x55, &signature.MapKey{})

Right, that's the expectation. Thanks for posting another option. Let's keep that in mind so we can later support a large number of acceptable runtimes if we want to.

kostko · 2019-10-16T08:39:50Z

go/worker/merge/committee/node.go

 		for _, result := range results {
 			ioRoots = append(ioRoots, result.IORoot)
 			stateRoots = append(stateRoots, result.StateRoot)
+
+			// Merge roothash messages.
+			// The rule is that at most one result can have sent roothash messages.


Added to list of slashing conditions for the transaction scheduler.

runtime/src/transaction/context.rs

pro-wh · 2019-10-16T18:44:53Z

cargo audit report shows no new problems compared to master. current problems, common with master:

memoffset 0.2.1 https://rustsec.org/advisories/RUSTSEC-2019-0011
protobuf 1.7.4 https://rustsec.org/advisories/RUSTSEC-2019-0003
protobuf 2.4.2 https://rustsec.org/advisories/RUSTSEC-2019-0003
smallvec 0.6.9 https://rustsec.org/advisories/RUSTSEC-2019-0009, https://rustsec.org/advisories/RUSTSEC-2019-0012
spin 0.5.0 https://rustsec.org/advisories/RUSTSEC-2019-0013
rust-crypto 0.2.36 https://rustsec.org/advisories/RUSTSEC-2016-0005

pro-wh force-pushed the pro-wh/feature/roothashmessage branch 3 times, most recently from 8a43a38 to 2af53df Compare October 4, 2019 00:24

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from 2af53df to 56d2f56 Compare October 5, 2019 00:53

pro-wh marked this pull request as ready for review October 7, 2019 18:21

pro-wh requested review from kostko, peterjgilbert and Yawning as code owners October 7, 2019 18:21

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from 56d2f56 to 976e4d9 Compare October 8, 2019 00:11

kostko reviewed Oct 8, 2019

View reviewed changes

go/roothash/api/block/header.go Outdated Show resolved Hide resolved

go/tendermint/apps/roothash/roothash.go Outdated Show resolved Hide resolved

go/tendermint/apps/roothash/roothash.go Show resolved Hide resolved

pro-wh force-pushed the pro-wh/feature/roothashmessage branch 3 times, most recently from a4b3ce8 to 1e9e0eb Compare October 9, 2019 00:00

pro-wh force-pushed the pro-wh/feature/roothashmessage branch 2 times, most recently from 45dbf40 to 1f16a93 Compare October 9, 2019 01:11

kostko reviewed Oct 9, 2019

View reviewed changes

go/roothash/api/block/header.go Outdated Show resolved Hide resolved

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from 1f16a93 to f5c67f6 Compare October 9, 2019 22:21

pro-wh mentioned this pull request Oct 9, 2019

Generalization of roothash message handlers #2247

Closed

kostko reviewed Oct 11, 2019

View reviewed changes

pro-wh added 4 commits October 11, 2019 13:16

runtime storage: eliminate double semicolon

9fa4b5d

go roothash: typo in tester

fe72eb4

go compute worker: avoid panic on shutdown

afae2a0

go tendermint: avoid panic on shutdown

494d989

pro-wh added 5 commits October 11, 2019 14:27

staking account update message prototype

f46221e

configure acceptable transfer peer policy from genesis

1107b28

testing framework for staking transfer

2f89d26

clean up dummy roothash message types

9ad9b90

go roothash: move mesasge types to new file

5bfe653

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from f5c67f6 to 611cf05 Compare October 11, 2019 21:28

go tendermint roothash: add wrapper around finalize steps

d2ca69e

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from 611cf05 to d2ca69e Compare October 11, 2019 23:18

pro-wh added 3 commits October 14, 2019 14:23

Merge remote-tracking branch 'origin/master' into HEAD

8c164a0

tests: add staking-arbitrary

1cd8e48

ci: build and upload staking-arbitrary

99a3f87

pro-wh force-pushed the pro-wh/feature/roothashmessage branch from eeffcba to 99a3f87 Compare October 15, 2019 01:07

pro-wh added 6 commits October 15, 2019 10:48

Merge remote-tracking branch 'origin/master' into HEAD

f5af0fa

go tendermint registry: log role on registration

8a4cd75

tests staking-arbitrary: fix import order

1800fc8

go test runner: make runtime scenario-specific

fec41be

go test runner: add roothash messages

91de707

go tendermint roothash: add log events

53aa6e7

pro-wh requested a review from kostko October 16, 2019 00:12

kostko approved these changes Oct 16, 2019

View reviewed changes

pro-wh mentioned this pull request Oct 16, 2019

byzantine node: be able to send roothash messages #2269

Open

pro-wh added 5 commits October 16, 2019 11:33

go test runner: don't log about client exiting

34e3b80

go staking: relocate LogEventGeneralAdjustment declaration

4b7c4cd

go tendermint staking: correct a description

b2d1d8b

runtime transaction: add doc comment to send_roothash_message

54b6b8d

Merge remote-tracking branch 'origin/master' into HEAD

546351e

pro-wh merged commit 163b84a into master Oct 16, 2019

pro-wh deleted the pro-wh/feature/roothashmessage branch October 16, 2019 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

messages from roothash mux app #2147

messages from roothash mux app #2147

pro-wh commented Sep 20, 2019 •

edited

Loading

kostko commented Oct 8, 2019

kostko left a comment

pro-wh commented Oct 8, 2019

pro-wh commented Oct 8, 2019

kostko commented Oct 8, 2019 •

edited

Loading

kostko commented Oct 8, 2019

pro-wh commented Oct 9, 2019

codecov bot commented Oct 9, 2019 •

edited

Loading

kostko left a comment

pro-wh commented Oct 16, 2019

kostko left a comment

kostko Oct 16, 2019

pro-wh Oct 16, 2019

kostko Oct 16, 2019

pro-wh Oct 16, 2019

pro-wh commented Oct 16, 2019 •

edited

Loading

messages from roothash mux app #2147

messages from roothash mux app #2147

Conversation

pro-wh commented Sep 20, 2019 • edited Loading

kostko commented Oct 8, 2019

kostko left a comment

Choose a reason for hiding this comment

pro-wh commented Oct 8, 2019

pro-wh commented Oct 8, 2019

kostko commented Oct 8, 2019 • edited Loading

kostko commented Oct 8, 2019

pro-wh commented Oct 9, 2019

codecov bot commented Oct 9, 2019 • edited Loading

Codecov Report

kostko left a comment

Choose a reason for hiding this comment

pro-wh commented Oct 16, 2019

kostko left a comment

Choose a reason for hiding this comment

kostko Oct 16, 2019

Choose a reason for hiding this comment

pro-wh Oct 16, 2019

Choose a reason for hiding this comment

kostko Oct 16, 2019

Choose a reason for hiding this comment

pro-wh Oct 16, 2019

Choose a reason for hiding this comment

pro-wh commented Oct 16, 2019 • edited Loading

pro-wh commented Sep 20, 2019 •

edited

Loading

kostko commented Oct 8, 2019 •

edited

Loading

codecov bot commented Oct 9, 2019 •

edited

Loading

pro-wh commented Oct 16, 2019 •

edited

Loading