Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core/consensus: add support for pointer values on the wire #1838

Merged
merged 8 commits into from
Mar 1, 2023

Conversation

corverroos
Copy link
Contributor

@corverroos corverroos commented Feb 28, 2023

Upgrades the QBFT wire protocol (without breaking backwards compatibility) by adding the whole value to the ConsensusMsg envelope and hashes of those values to the QBFTMsg itself. This will allow us to remove the values from the QBFT messages, which will reduce network bandwidth usage by a lot (90% for large values; beacon blocks).

This increases the network bandwidth for next release v0.14, but once we remove support for v0.13, it will reduce it by a lot.

category: feature
ticket: #1552

@@ -73,150 +69,28 @@ func TestSigning(t *testing.T) {
require.False(t, ok)
}

func TestBackwardsCompatibility(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this tests 2 updates ago, we already removed support, can remove now.

repeated QBFTMsg justification = 2;
QBFTMsg msg = 1;
repeated QBFTMsg justification = 2;
repeated google.protobuf.Any values = 3;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the values are added here and can therefore be removed from QBFTMsg above once we remove support for v0.13.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: can we add a comment for future reference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

@codecov
Copy link

codecov bot commented Feb 28, 2023

Codecov Report

Patch coverage: 53.39% and project coverage change: +0.10 🎉

Comparison is base (61348a9) 55.34% compared to head (4b8044e) 55.45%.

❗ Current head 4b8044e differs from pull request most recent head b71b193. Consider uploading reports for the commit b71b193 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1838      +/-   ##
==========================================
+ Coverage   55.34%   55.45%   +0.10%     
==========================================
  Files         170      170              
  Lines       22332    22254      -78     
==========================================
- Hits        12360    12340      -20     
+ Misses       8385     8337      -48     
+ Partials     1587     1577      -10     
Impacted Files Coverage Δ
app/app.go 60.89% <0.00%> (ø)
testutil/genchangelog/main.go 14.44% <0.00%> (ø)
core/consensus/msg.go 54.34% <35.48%> (+1.06%) ⬆️
dkg/disk.go 58.26% <50.00%> (ø)
core/consensus/component.go 59.82% <53.33%> (-1.69%) ⬇️
core/consensus/transport.go 75.00% <70.27%> (+10.41%) ⬆️
eth2util/keystore/keystore.go 50.90% <100.00%> (ø)
app/vmock.go 71.28% <0.00%> (-3.08%) ⬇️
core/parsigdb/memory.go 48.09% <0.00%> (-1.53%) ⬇️
core/scheduler/scheduler.go 76.20% <0.00%> (-1.15%) ⬇️
... and 167 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@gsora gsora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General approve because it looks good.

I wonder if we could get rid of the *anypb.Any and deal with copies instead.

@@ -545,3 +568,22 @@ func newRoundTimer(round int64) (<-chan time.Time, func()) {
timer := time.NewTimer(roundStart + (time.Duration(round) * roundIncrease))
return timer.C, func() { timer.Stop() }
}

func valuesByHash(values []*anypb.Any) (map[[32]byte]*anypb.Any, error) {
resp := make(map[[32]byte]*anypb.Any)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this function return error if values is empty?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, empty values result in a empty map, this is required for legacy v0.13 support.

repeated QBFTMsg justification = 2;
QBFTMsg msg = 1;
repeated QBFTMsg justification = 2;
repeated google.protobuf.Any values = 3;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: can we add a comment for future reference?

@corverroos
Copy link
Contributor Author

@gsora I do not think we should get rid of anypb.Any in this PR, that is a bigger refactor that addresses a different problem. Also, I think Any is working fine for our use-case. You'd have to explain your concerns a bit clearer as I do not see any problem at this point.

@gsora
Copy link
Collaborator

gsora commented Feb 28, 2023

Sorry, didn't mean to imply that Any should go away, rather if we could replace a map of pointers with a map of objects.

I'm nitpicking though, because I really would love to avoid pointers when possible.

@corverroos
Copy link
Contributor Author

Since the Anys wrap protobufs, the wrapped objects are still pointers unfortunately.

@corverroos corverroos added the merge when ready Indicates bulldozer bot may merge when all checks pass label Mar 1, 2023
@obol-bulldozer obol-bulldozer bot merged commit dfabc1c into main Mar 1, 2023
@obol-bulldozer obol-bulldozer bot deleted the corver/qbftpointers branch March 1, 2023 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge when ready Indicates bulldozer bot may merge when all checks pass
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants