Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[double-sign] Provide proof of double sign in slash record sent to beaconchain #2253

Merged
merged 141 commits into from
Feb 28, 2020
Merged

[double-sign] Provide proof of double sign in slash record sent to beaconchain #2253

merged 141 commits into from
Feb 28, 2020

Conversation

fxfactorial
Copy link
Contributor

@fxfactorial fxfactorial commented Feb 12, 2020

Large PR which implements slashing as result of a double-signing event.

cmd/harmony/main.go Outdated Show resolved Hide resolved
@@ -217,9 +217,21 @@ func setupLegacyNodeAccount() error {
_, initialAccount = genesisShardingConfig.FindAccount(pubKey.SerializeToHexStr())
}

if initialAccount == nil {
const crtValdidatorBLS = "be23bc3c93fe14a25f3533feee1cff1c60706845a4907c5df58bc19f5d1760bfff06fe7c9d1f596b18fdf529e0508e0a"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up

@@ -0,0 +1,25 @@
package main
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put into /test if you want to keep it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do you build into binary if put under test? i mean, will go_exectuable_build.sh build it and put into bin?

@@ -129,6 +129,8 @@ type Consensus struct {
lastBlockReward *big.Int
// Have a dedicated reader thread pull from this chan, like in node
SlashChan chan slash.Record
// only during testing
DoDoubleSign bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup

consensusMsg = consensus.populateMessageFields(
message.GetConsensus(), blockHash[:],
)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should you also change the "case msg_pb.MessageType_COMMIT:" part?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it currently is handled correctly

@@ -398,6 +400,9 @@ func (b *APIBackend) GetTotalStakingSnapshot() *big.Int {
stakes := big.NewInt(0)
for i := range candidates {
validator, _ := b.hmy.BlockChain().ReadValidatorInformation(candidates[i])
if !effective.IsEligibleForEPOSAuction(validator) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add this condition here? What is "GetTotalStakingSnapshot" used for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because otherwise you are counting the stake of banned or active=false validators for the median stake

}
// candidate.ConflictingBallots

// TODO Why this one printng have 00000000 for signature? something wrong earlier
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these todos are launch blocking, better create issues for better tracking.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put under "Staking" project.

cmd/harmony/main.go Outdated Show resolved Hide resolved
consensus/construct.go Outdated Show resolved Hide resolved
consensus/leader.go Show resolved Hide resolved
core/state/statedb.go Outdated Show resolved Hide resolved
internal/configs/sharding/testnet.go Outdated Show resolved Hide resolved
node/node_genesis.go Show resolved Hide resolved
node/double_signing.go Show resolved Hide resolved
staking/slash/double-sign.go Show resolved Hide resolved
staking/slash/double-sign.go Outdated Show resolved Hide resolved
staking/slash/double-sign.go Outdated Show resolved Hide resolved

cls = append(cls, candidates...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no dedup?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think technically not needed since I clear out all the slashing candidates in end writeblockwithstate

core/staking_verifier.go Outdated Show resolved Hide resolved
internal/chain/engine.go Outdated Show resolved Hide resolved
node/node_genesis.go Show resolved Hide resolved
@@ -300,10 +306,56 @@ func (w *Worker) IncomingReceipts() []*types.CXReceiptsProof {
return w.current.incxs
}

// CollectAndVerifySlashes ..
func (w *Worker) CollectAndVerifySlashes() error {
allSlashing, err := w.chain.ReadPendingSlashingCandidates()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do you remove the one that's proposed from pending?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i remove them all in writeblockwithstate,

			if err := bc.UpdateBlockRewardAccumulator(
				batch, payout, block.Number().Uint64(),
			); err != nil {
				return NonStatTy, err
			}
			if err := bc.DeletePendingSlashingCandidates(); err != nil {
				return NonStatTy, err
			}

staking/slash/double-sign.go Show resolved Hide resolved
staking/slash/double-sign.go Show resolved Hide resolved
@fxfactorial fxfactorial changed the title [wip][double-sign] Provide proof of double sign in slash record sent to beaconchain [double-sign] Provide proof of double sign in slash record sent to beaconchain Feb 28, 2020
@fxfactorial fxfactorial merged commit c4d744e into harmony-one:master Feb 28, 2020
@fxfactorial fxfactorial deleted the double-sign branch February 28, 2020 01:30
@fxfactorial fxfactorial restored the double-sign branch February 29, 2020 06:35
flicker-harmony pushed a commit to flicker-harmony/harmony that referenced this pull request Mar 3, 2020
parent eb93305
author flicker-harmony <[email protected]> 1583258969 +0300
committer flicker-harmony <[email protected]> 1583258969 +0300

Resolve conflict

# This is the commit message harmony-one#5:

[double-sign] Provide proof of double sign in slash record sent to beaconchain (harmony-one#2253)

* [double-sign] Commit changes in consensus needed for double-sign

* [double-sign] Leader captures when valdator double signs, broadcasts to beaconchain

* [slash] Add quick iteration tool for testing double-signing

* [slash] Add webhook example

* [slash] Add http server for hook to trigger double sign behavior

* [double-sign] Use bin/trigger-double-sign to cause a double-sign

* [double-sign] Full feedback loop working

* [slash] Thread through the slash records in the block proposal step

* [slash] Compute the slashing rate

* [double-sign] Generalize yaml malicious for many keys

* [double-sign][slash] Modify data structures, verify via webhook handler

* [slash][double-sign] Find one address of bls public key signer, seemingly settle on data structures

* [slash] Apply to state slashing for double signing

* [slash][double-sign] Checkpoint for working code that slashes on beaconchain

* [slash] Keep track of the total slash and total reporters reward

* [slash] Dump account state before and after the slash

* [slash] Satisfy Travis

* [slash][state] Apply slash to the snapshot at beginning of epoch, now need to capture also the new delegates

* [slash] Capture the unique new delegations since snapshot as well

* [slash] Filter undelegation by epoch of double sign

* [slash] Add TODO of correctness needed in slash needs on off-chain data

* [rpc] Fix closure issue on shardID

* [slash] Add delegator to double-sign testing script

* [slash] Expand crt-validator.sh with commenting printfs and make delegation

* [slash] Finish track payment of leftover slash debt after undelegation runs out

* [slash] Now be explicit about error wrt delegatorSlashApply

* [slash] Capture specific sanity check on slash paidoff

* [slash] Track slash from undelegation piecemeal

* [slash][delegation] Named slice types, .String()

* [slash] Do no RLP encode twice, once is enough

* [slash] Remove special case of validators own delegation

* [slash] Refactor approach to slash state application

* [slash] Begin expanding out Verify

* [slash] Slash on snapshot delegations, not current

* [slash] Fix Epoch Cmp

* [slash] Third iteration on slash logic

* [slash] Use full slash amount

* [slash] More log, whitespace

* [slash] Remove Println, add log

* [slash] Remove debug Println

* [slash] Add record in unit test

* [slash] Build Validator snapshot, current. Fill out slash record

* [slash] Need to get RLP dump of a header to use in test

* [slash] Factor out double sign test constants

* [slash] Factor out common for validator, stub out slash application, finish out deserialization setup

* [slash] Factor out data structure creation because of var lexical scoping

* [slash] Seem to have pipeline of unit test e2e executing

* [slash] Add expected snitch, slash amounts

* [slash] Checkpoint

* [slash] Unit test correctly checks case of validator own stake which could drop below 1 ONE in slashing

* [config] add double-sign testnet config (#1)

Signed-off-by: Leo Chen <[email protected]>

* [slash] Commit for as is code & data of current dump.json

* [slash] Order of state operation not correct in test, hence bad results, thank you dlv

* [slash] Add snapshot state dump

* [slash] Pay off slash of validator own delegation correctly

* [slash] Pay off slash debt with special case for min-self

* [slash] Pass first scenario conclusively

* [slash] 2% slash passes unit test for own delegation and external

* [slash] Parameterize unit test to easily test .02 vs .80 slash

* [slash] Handle own delegation correctly at 80% slash

* [slash] Have 80% slash working with external delegator

* [slash] Remove debug code from slash

* [slash] Adjust Apply signature, test again for 2% slash

* [slash] Factor out scenario in testing so can test 2% and 80% at same time

* [slash] Correct balance deduction on plan delegation

* [slash] Mock out ChainReader for TestVerify

* [slash] Small surface area interface, now feedback loop for verify

* [slash] Remove development json

* [slash] trigger-double-sign consumes yaml

* [slash] Remove dead code

* [slash][test] Factor ValidatorWrapper into scenario

* [slash][test] Add example from local-testing dump - caution might be off

* [slash] Factor out mutation of slashDebt

* [slash][test] Factor out tests so can easily load test-case from bytes

* [slash] Fix payment mistake in validator own delegation wrt min-self-delgation respected

* [slash] Satisfy Travis

* [slash] Begin cleanup of PR

* [slash] Apply slash from header to Finalize via state processor

* [slash] Productionize code, Println => logs; adjust slash picked in newblock

* [slash] Need pointer for rlp.Decode

* [slash] ValidatorInformation use full wrapper

* Fix median stake

* [staking] Adjust MarshalJSON for Validator, Wrapper

* Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279)

* Refactor offchain data; Add epoch to ValidatorSnapshot

* Make block onchain/offchain data commit atomically

* [slash][committee] Set .Active to false on double sign, do not consider banned or inactive for committee assignment

* [effective] VC eligible.go

* [consensus] Redundant field in printf

* [docker] import-ks for a dev account

* [slash] Create BLS key for dockerfile and crt-validator.sh

* [slash][docker] Easy deployment of double-sign testing

* [docker] Have slash work as single docker command

* [rpc] Fix median-stake RPC

* [slash] Update webhook with default docker BLS key

* [docker][slash] Fresh yaml copy for docker build, remove dev code in main.go

* [slash] Remove helper binary, commented out code, change to local config

* [params] Factor out test genesis value

* Add shard checking to Tx-Pool & correct blacklist (harmony-one#2301)

* [core] Fix blacklist & add shardID check

* [staking + node + cmd] Fix blacklist & add shardID check

* [slash] Adjust to PR comments part 1

* [docker] Use different throw away funded account

* [docker] Create easier testing for delegation with private keys

* [docker] Update yaml

* [slash] Remove special case for slashing validator own delegation wrt min-self-delegate

* [docker] Install nano as well

* [slash] Early error if banned

* [quorum] Expose earning account in decider marshal json

* Revert "Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279)"

This reverts commit 9ffbf68.

* [slash] Add non-sanity check way to update validator

* [reward] Increase percision on percentage in schedule

* [slash] Adjust logs

* [committee] Check eligibility of validator before doing sanity check

* [slash] Update docker

* [slash] Move create validator script to test

* [slash] More log

* [param] Make things faster

* [slash][off-chain] Clear out slashes from pending in writeblockwithstate

* [cross-link] Log is not error, just info

* [blockchain] Not necessary to guard DeletePendingSlashingCandidates

* [slash][consensus] Use plain []byte for signature b/c bls.Sign has private impl fields, rlp does not encode that

* [slash][test] Use faucet as sender, assume user imported

* [slash] Test setup

* [slash] reserve error for real error in logs

* [slash][availability] Apply availability correct, bump signing count each block

* [slash][staking] Consider banned field in sanity check, pay snitch only half of what was actually slashed

* [slash] Pay as much as can

* [slash] use right nowAmt

* [slash] Take away from rewards as well

* [slash] iterate faster

* [slash] Remove dev based timing

* [slash] Add more log, sanity check incoming slash records, only count external for slash rate

* [availability][state] Adjust signature of ValidatorWrapper wrt state, filter out for staked validators, correct availaibility measure on running counters

* [availability] More log

* [slash] Simply pre slash erra slashing

* [slash] Remove development code

* [slash] Use height from recvMsg, todo on epoch

* [staking] Not necessary to touch LastEpochInCommittee in staking_verifier

* [slash] Undo ds in endpoint pattern config

* [slash] Add TODO and log when delegation becomes 0 b/c slash debt payment

* [slash] Abstract staked validators from shard.State into type, set slash rate based BLSKey count

Co-authored-by: Leo Chen <[email protected]>
Co-authored-by: flicker-harmony <[email protected]>
Co-authored-by: Rongjian Lan <[email protected]>
Co-authored-by: Daniel Van Der Maden <[email protected]>

# This is the commit message harmony-one#7:

Fix last continuous crosslink logic (harmony-one#2316)

* Fix last continuous crosslink logic

* fix lint

* Add comment

# This is the commit message harmony-one#8:

new profile for OSTN

# This is the commit message harmony-one#9:

[project] Remove leftover changes to node.sh from slashing work (harmony-one#2326)
denniswon pushed a commit to denniswon/harmony that referenced this pull request Mar 7, 2020
…aconchain (harmony-one#2253)

* [double-sign] Commit changes in consensus needed for double-sign

* [double-sign] Leader captures when valdator double signs, broadcasts to beaconchain

* [slash] Add quick iteration tool for testing double-signing

* [slash] Add webhook example

* [slash] Add http server for hook to trigger double sign behavior

* [double-sign] Use bin/trigger-double-sign to cause a double-sign

* [double-sign] Full feedback loop working

* [slash] Thread through the slash records in the block proposal step

* [slash] Compute the slashing rate

* [double-sign] Generalize yaml malicious for many keys

* [double-sign][slash] Modify data structures, verify via webhook handler

* [slash][double-sign] Find one address of bls public key signer, seemingly settle on data structures

* [slash] Apply to state slashing for double signing

* [slash][double-sign] Checkpoint for working code that slashes on beaconchain

* [slash] Keep track of the total slash and total reporters reward

* [slash] Dump account state before and after the slash

* [slash] Satisfy Travis

* [slash][state] Apply slash to the snapshot at beginning of epoch, now need to capture also the new delegates

* [slash] Capture the unique new delegations since snapshot as well

* [slash] Filter undelegation by epoch of double sign

* [slash] Add TODO of correctness needed in slash needs on off-chain data

* [rpc] Fix closure issue on shardID

* [slash] Add delegator to double-sign testing script

* [slash] Expand crt-validator.sh with commenting printfs and make delegation

* [slash] Finish track payment of leftover slash debt after undelegation runs out

* [slash] Now be explicit about error wrt delegatorSlashApply

* [slash] Capture specific sanity check on slash paidoff

* [slash] Track slash from undelegation piecemeal

* [slash][delegation] Named slice types, .String()

* [slash] Do no RLP encode twice, once is enough

* [slash] Remove special case of validators own delegation

* [slash] Refactor approach to slash state application

* [slash] Begin expanding out Verify

* [slash] Slash on snapshot delegations, not current

* [slash] Fix Epoch Cmp

* [slash] Third iteration on slash logic

* [slash] Use full slash amount

* [slash] More log, whitespace

* [slash] Remove Println, add log

* [slash] Remove debug Println

* [slash] Add record in unit test

* [slash] Build Validator snapshot, current. Fill out slash record

* [slash] Need to get RLP dump of a header to use in test

* [slash] Factor out double sign test constants

* [slash] Factor out common for validator, stub out slash application, finish out deserialization setup

* [slash] Factor out data structure creation because of var lexical scoping

* [slash] Seem to have pipeline of unit test e2e executing

* [slash] Add expected snitch, slash amounts

* [slash] Checkpoint

* [slash] Unit test correctly checks case of validator own stake which could drop below 1 ONE in slashing

* [config] add double-sign testnet config (#1)

Signed-off-by: Leo Chen <[email protected]>

* [slash] Commit for as is code & data of current dump.json

* [slash] Order of state operation not correct in test, hence bad results, thank you dlv

* [slash] Add snapshot state dump

* [slash] Pay off slash of validator own delegation correctly

* [slash] Pay off slash debt with special case for min-self

* [slash] Pass first scenario conclusively

* [slash] 2% slash passes unit test for own delegation and external

* [slash] Parameterize unit test to easily test .02 vs .80 slash

* [slash] Handle own delegation correctly at 80% slash

* [slash] Have 80% slash working with external delegator

* [slash] Remove debug code from slash

* [slash] Adjust Apply signature, test again for 2% slash

* [slash] Factor out scenario in testing so can test 2% and 80% at same time

* [slash] Correct balance deduction on plan delegation

* [slash] Mock out ChainReader for TestVerify

* [slash] Small surface area interface, now feedback loop for verify

* [slash] Remove development json

* [slash] trigger-double-sign consumes yaml

* [slash] Remove dead code

* [slash][test] Factor ValidatorWrapper into scenario

* [slash][test] Add example from local-testing dump - caution might be off

* [slash] Factor out mutation of slashDebt

* [slash][test] Factor out tests so can easily load test-case from bytes

* [slash] Fix payment mistake in validator own delegation wrt min-self-delgation respected

* [slash] Satisfy Travis

* [slash] Begin cleanup of PR

* [slash] Apply slash from header to Finalize via state processor

* [slash] Productionize code, Println => logs; adjust slash picked in newblock

* [slash] Need pointer for rlp.Decode

* [slash] ValidatorInformation use full wrapper

* Fix median stake

* [staking] Adjust MarshalJSON for Validator, Wrapper

* Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279)

* Refactor offchain data; Add epoch to ValidatorSnapshot

* Make block onchain/offchain data commit atomically

* [slash][committee] Set .Active to false on double sign, do not consider banned or inactive for committee assignment

* [effective] VC eligible.go

* [consensus] Redundant field in printf

* [docker] import-ks for a dev account

* [slash] Create BLS key for dockerfile and crt-validator.sh

* [slash][docker] Easy deployment of double-sign testing

* [docker] Have slash work as single docker command

* [rpc] Fix median-stake RPC

* [slash] Update webhook with default docker BLS key

* [docker][slash] Fresh yaml copy for docker build, remove dev code in main.go

* [slash] Remove helper binary, commented out code, change to local config

* [params] Factor out test genesis value

* Add shard checking to Tx-Pool & correct blacklist (harmony-one#2301)

* [core] Fix blacklist & add shardID check

* [staking + node + cmd] Fix blacklist & add shardID check

* [slash] Adjust to PR comments part 1

* [docker] Use different throw away funded account

* [docker] Create easier testing for delegation with private keys

* [docker] Update yaml

* [slash] Remove special case for slashing validator own delegation wrt min-self-delegate

* [docker] Install nano as well

* [slash] Early error if banned

* [quorum] Expose earning account in decider marshal json

* Revert "Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279)"

This reverts commit 9ffbf68.

* [slash] Add non-sanity check way to update validator

* [reward] Increase percision on percentage in schedule

* [slash] Adjust logs

* [committee] Check eligibility of validator before doing sanity check

* [slash] Update docker

* [slash] Move create validator script to test

* [slash] More log

* [param] Make things faster

* [slash][off-chain] Clear out slashes from pending in writeblockwithstate

* [cross-link] Log is not error, just info

* [blockchain] Not necessary to guard DeletePendingSlashingCandidates

* [slash][consensus] Use plain []byte for signature b/c bls.Sign has private impl fields, rlp does not encode that

* [slash][test] Use faucet as sender, assume user imported

* [slash] Test setup

* [slash] reserve error for real error in logs

* [slash][availability] Apply availability correct, bump signing count each block

* [slash][staking] Consider banned field in sanity check, pay snitch only half of what was actually slashed

* [slash] Pay as much as can

* [slash] use right nowAmt

* [slash] Take away from rewards as well

* [slash] iterate faster

* [slash] Remove dev based timing

* [slash] Add more log, sanity check incoming slash records, only count external for slash rate

* [availability][state] Adjust signature of ValidatorWrapper wrt state, filter out for staked validators, correct availaibility measure on running counters

* [availability] More log

* [slash] Simply pre slash erra slashing

* [slash] Remove development code

* [slash] Use height from recvMsg, todo on epoch

* [staking] Not necessary to touch LastEpochInCommittee in staking_verifier

* [slash] Undo ds in endpoint pattern config

* [slash] Add TODO and log when delegation becomes 0 b/c slash debt payment

* [slash] Abstract staked validators from shard.State into type, set slash rate based BLSKey count

Co-authored-by: Leo Chen <[email protected]>
Co-authored-by: flicker-harmony <[email protected]>
Co-authored-by: Rongjian Lan <[email protected]>
Co-authored-by: Daniel Van Der Maden <[email protected]>
}
currentNode.NodeConfig.WebHooks.DoubleSigning = config
}
currentNode := node.New(myHost, currentConsensus, chainDBFactory, blacklist, true)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LeoHChen can you comment on changing *isArchival to true at node.New(myHost, currentConsensus, chainDBFactory, blacklist, *isArchival)? cc @rlan35

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants