Deneb Support #324

MatheusFranco99 · 2023-12-20T09:48:33Z

Major changes

Added logical support to Deneb's beacon block in consensus data.
Note: When submitting a proposer duty output (VersionedProposal or VersionedBlindedProposal) to the Beacon node, Deneb uses:
- VersionedBlindedProposal: a BlindedBeaconBlock, similarly to other forks.
- VersionedProposal: BlockContents, differently from older forks which used BeaconBlocks. Nonetheless, the validator signature is still over the beacon block (BlockContents.Block), i.e. the signing root is kept the same.

Minor changes

Updated Attestantio lib from 0.16.3 to 0.19.7. Purpose: use up-to-date Deneb spec types.
Updated Fastssz from 0.1.2 to 0.1.3 as a consequence of the Attestantio update.
Deneb testing utils. To-Do: define Deneb fork epoch.
Added type tests.
Remove all Bellatrix objects and use Deneb.
Update ssz-max sizes for ConsensusData and FullData fields, since Deneb uses a BlockContents object which contains a Beacon block, KZGProofs (288 bytes) and Blobs of 786432 bytes.

SSZ generation notes

Fastssz 0.1.3 is now deterministic. Because of this, when you run make generate-ssz you may see new logs INFO: Skipped ssz generated object:.
Fastssz 0.1.3 seems to have an error which occurs in the consensus_data_encoding.go file. It imports an unused library. For that, we created an issue in Fastssz which remains unsolved. As a temporary arrangement, we added, in the generate.go in ./types, a command to remove the unused import.

Attestantio update notes

The default DataVersion used to be phase0. This was was changed to become unknown. Therefore, some testing utils that didn't define the DataVersion in the ConsensusData had to be updated.

Leftovers To-Do

Set the correct Deneb fork epoch.

References

Deneb Consensus-Spec

GalRogozinski · 2023-12-24T08:54:22Z

go.mod

@@ -47,8 +47,8 @@ require (
 	go.uber.org/atomic v1.10.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.24.0 // indirect
-	golang.org/x/crypto v0.7.0 // indirect
-	golang.org/x/sys v0.7.0 // indirect
+	golang.org/x/crypto v0.10.0 // indirect


This less matters for us in the spec project, but when ssv is forced to dot he same update it may affect performance as we learnt for the RSA debacle

That's true. We shall do performance tests when included. Just to remember, for the RSA case, we're actually using microsoft's openssl so it's less of an issue.

GalRogozinski · 2023-12-24T12:34:56Z

types/spectest/tests/ssz/capella_marshaling_validation.go

+		panic(err.Error())
+	}
+
+	return &SSZSpecTest{


I think we only created an SSZSpecTest for Withdrawals because we had some custom logic for it (don't remember why). I think because the latest go-eth2-client release at the time didn't have it.
I think if BeaconBlock now supports withdrawals (and also Deneb blobs) we can remove the old withdrawals test and code (maybe in separate PR after this because we have enough code here)

Anyhow, you are aware of my evil plan with value check that will make all this logic/tests unnecessary 😈

#324 (comment)

GalRogozinski · 2023-12-24T12:37:03Z

types/ssz.go

+func (b SSZBlobZGCommitments) GetTree() (*ssz.Node, error) {
+	return ssz.ProofTree(b)
+}
+
+func (b SSZBlobZGCommitments) HashTreeRootWith(hh ssz.HashWalker) error {
+	// taken from https://github.com/attestantio/go-eth2-client/blob/a05485e0e75749f2b6912db2972a35ec2ec37c3b/spec/deneb/beaconblockbody_ssz.go#L577C3-L586C49
+	if size := len(b); size > 4096 {
+		err := ssz.ErrListTooBigFn("BeaconBlockBody.BlobKZGCommitments", size, 4096)
+		return err
+	}
+	subIndx := hh.Index()
+	for _, i := range b {
+		hh.PutBytes(i[:])
+	}
+	numItems := uint64(len(b))
+	hh.MerkleizeWithMixin(subIndx, numItems, 4096)
+	return nil
+}


did you do this because we did the same thing with Withdrawals?

I think we did this to Withdrawals because there wasn't a release available at the time (am I correct @olegshmuelov ?)

If the updated go-eth2-client has this logic then I don't think we need to copy it..

Please see previous discussion
#234 (comment)

GalRogozinski · 2023-12-28T09:52:28Z

There seems to be some epoch activation epochs here

Not sure if it is final but maybe let's put this in and later change it so we can all approve this PR

MatheusFranco99 · 2023-12-29T17:31:17Z

@GalRogozinski If I update it, I start getting:

panic: can't start new duty runner instance for duty: input data invalid: duty invalid: duty epoch is into far future

because of the dutyValueCheck.

GalRogozinski · 2024-01-01T11:32:28Z

types/spectest/tests/consensusdata/proposer/versioned_block_spectest.go

 		case spec.DataVersionCapella:
 			require.NotNil(t, vBlk.Capella)
 			blkSSZ, err = vBlk.Capella.MarshalSSZ()
 			require.NoError(t, err)
+		case spec.DataVersionDeneb:


if you scroll up you see a comment for issue 219. IMO you can delete the comment and close the issue

GalRogozinski · 2024-01-03T09:41:38Z

Closes #219

types/testingutils/beacon_node.go

GalRogozinski · 2024-01-16T17:21:05Z

types/testingutils/beacon_node_versioned.go

+		if vBlk.Deneb.Block == nil {
+			panic("empty block")
+		}
+		return &deneb.SignedBeaconBlock{


Question: there is also api/v1/deneb/signedblindedbeaconblock.go
shouldn't this one be used here?

Well noticed. I'm fixing it now. At first, I used version.SignedBeaconBlock because it was already being used for the blinded proposer beacon roots, strangely enough. But the correct one should be the SignedBlindedBeaconBlock indeed.

moshe-blox · 2024-01-17T14:51:18Z

types/consensus_data.go

-	// we can upper limit transactions to 2^21, together with the rest of the data 2*2^21 = 2^22 = 4,194,304 bytes
-	// Exmplanation on why transaction sizes are so big https://github.com/ethereum/consensus-specs/pull/2686
-	DataSSZ []byte `ssz-max:"4194304"` // 2^22
+	// we can upper limit transactions to 2^21, together with the rest of the data 1315964 + 2^21  = 3,413,116 bytes


Why did the max decrease from ~4MB to ~3MB (2*2^21 to 2^21)? If anything, the introduction of blobs should have increased our ssz-max, no?

BTW, they're thinking of increasing gas size from 30M to 40M very soon, we should probably adapt our calculations (if needed).

I think we could maybe check what was Prysm's approach to this. Prior to receiving blocks from peers, they must do some sort of validation on the size, I guess.

See this comment where we corrected a previous mistake regarding attestations:
#324 (comment)

checkout the commented python script

they're thinking of increasing gas size from 30M to 40M very soon
When they are done thinking we'll fix this

it will only be on the next hard fork (best case) so there is time

they're thinking of increasing gas size from 30M to 40M very soon
When they are done thinking we'll fix this

it will only be on the next hard fork (best case) so there is time

I think maybe we'd like to support a probable gas increase without having to fork?

This reverts commit f2d5899.

This reverts commit 0a559e2.

* Revert "Revert "Deneb Support (#324)"" This reverts commit 0a559e2. * use custom old encodings * generate-jsons * Add phase0 version * Bellatrix -> Capella on a single test * Generate jsons * Revert Capella -> Deneb on a single test --------- Co-authored-by: MatheusFranco99 <[email protected]>

…compatibility (#362) * Revert "Deneb Support (#324)" This reverts commit f2d5899. * Add jsons (#357) * create jsons * compare with jsons * generate jsons for qbft signed message * compare qbft signed message with json state * fix lint error * generate jsons for create msg test * generate jsons for create msg test * use correct test type * use correct test name * Return deneb with old encoding (#360) * Revert "Revert "Deneb Support (#324)"" This reverts commit 0a559e2. * use custom old encodings * generate-jsons * Add phase0 version * Bellatrix -> Capella on a single test * Generate jsons * Revert Capella -> Deneb on a single test --------- Co-authored-by: MatheusFranco99 <[email protected]> --------- Co-authored-by: MatheusFranco99 <[email protected]>

Add beacon block option in consensus data

5983a53

MatheusFranco99 self-assigned this Dec 20, 2023

MatheusFranco99 added 6 commits December 20, 2023 07:57

Update Attestation and fastssz. Generate new ssz [skip ci]

70dd2d2

Add KZGCommitment SSZ

d8362d0

Add unused generated import just to show in the FastSSZ issue

031add0

Remove unused import using the goimports tool

5905462

Add Deneb beacon node for testing

7337929

Add beacon node testing util

a7018ef

GalRogozinski mentioned this pull request Dec 21, 2023

remove bellatrix objects #173

Open

MatheusFranco99 added 4 commits December 22, 2023 09:36

Add DataVersion due to change of default value

22c5d9f

Add type ssz tests

eba0818

Remove Bellatrix objects and use Deneb [skip ci]

8f8acae

Generate new tests

339eaa6

MatheusFranco99 marked this pull request as ready for review December 23, 2023 14:57

MatheusFranco99 requested review from GalRogozinski and alonmuroch December 23, 2023 14:58

GalRogozinski reviewed Dec 24, 2023

View reviewed changes

Revert ZGCommitments SSZ

d6611b1

MatheusFranco99 requested review from GalRogozinski and olegshmuelov December 27, 2023 13:32

GalRogozinski requested a review from y0sher January 1, 2024 09:12

GalRogozinski reviewed Jan 1, 2024

View reviewed changes

Drop comment about issue

3d553a7

MatheusFranco99 requested a review from GalRogozinski January 2, 2024 08:31

olegshmuelov reviewed Jan 11, 2024

View reviewed changes

types/testingutils/beacon_node.go Outdated Show resolved Hide resolved

olegshmuelov requested changes Jan 11, 2024

View reviewed changes

types/testingutils/beacon_node.go Outdated Show resolved Hide resolved

types/testingutils/beacon_node.go Outdated Show resolved Hide resolved

Drop unused test variables

49e2bdd

MatheusFranco99 and others added 8 commits January 16, 2024 09:53

SSZ-max adjustment

69343a4

Use fixed capellaBlock const

091bcbf

Generate tests

d99feb8

revert to bellatrix

d56eaa9

wrong duty type fix

88bab4d

re generate json tests

8c10119

revert sync committee contribution version to bellatrix

5db345b

regenerate tests

8dbbe6d

GalRogozinski reviewed Jan 16, 2024

View reviewed changes

MatheusFranco99 added 3 commits January 17, 2024 07:26

Fix testing signed blinded block

136f0ea

Update ssz-max. Add python code

9c307b1

Use correct prater fork epoch for deneb

14892ee

y0sher approved these changes Jan 17, 2024

View reviewed changes

GalRogozinski requested review from olegshmuelov, y0sher and moshe-blox January 17, 2024 12:48

y0sher approved these changes Jan 17, 2024

View reviewed changes

moshe-blox reviewed Jan 17, 2024

View reviewed changes

Adjust szz-max size

e7e9a74

moshe-blox approved these changes Jan 18, 2024

View reviewed changes

olegshmuelov approved these changes Jan 18, 2024

View reviewed changes

GalRogozinski merged commit f2d5899 into main Jan 21, 2024
2 checks passed

GalRogozinski deleted the deneb-support branch January 21, 2024 13:51

moshe-blox mentioned this pull request Jan 30, 2024

Revert changes in HashTreeRoot and SSZ encoding for backwards-compatibility #354

Closed

GalRogozinski added a commit that referenced this pull request Jan 31, 2024

Revert "Deneb Support (#324)"

0a559e2

This reverts commit f2d5899.

GalRogozinski added a commit that referenced this pull request Feb 1, 2024

Revert "Revert "Deneb Support (#324)""

de6cc55

This reverts commit 0a559e2.

moshe-blox mentioned this pull request Feb 1, 2024

Revert breaking encoding changes #361

Closed

GalRogozinski mentioned this pull request Feb 1, 2024

Revert changes in HashTreeRoot and SSZ encoding for backwards-compatibility #362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deneb Support #324

Deneb Support #324

MatheusFranco99 commented Dec 20, 2023 •

edited

Loading

GalRogozinski Dec 24, 2023

MatheusFranco99 Dec 27, 2023

GalRogozinski Dec 24, 2023

olegshmuelov Dec 24, 2023

GalRogozinski Dec 24, 2023

olegshmuelov Dec 24, 2023

GalRogozinski commented Dec 28, 2023 •

edited

Loading

MatheusFranco99 commented Dec 29, 2023

GalRogozinski Jan 1, 2024

GalRogozinski commented Jan 3, 2024

GalRogozinski Jan 16, 2024

MatheusFranco99 Jan 17, 2024

moshe-blox Jan 17, 2024 •

edited

Loading

GalRogozinski Jan 17, 2024

GalRogozinski Jan 17, 2024

moshe-blox Jan 18, 2024

Deneb Support #324

Deneb Support #324

Conversation

MatheusFranco99 commented Dec 20, 2023 • edited Loading

Major changes

Minor changes

SSZ generation notes

Attestantio update notes

Leftovers To-Do

References

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GalRogozinski commented Dec 28, 2023 • edited Loading

MatheusFranco99 commented Dec 29, 2023

Choose a reason for hiding this comment

GalRogozinski commented Jan 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moshe-blox Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatheusFranco99 commented Dec 20, 2023 •

edited

Loading

GalRogozinski commented Dec 28, 2023 •

edited

Loading

moshe-blox Jan 17, 2024 •

edited

Loading