approval-distribution: batch assignment and approval vote send

[sandreim]

Metrics revealed that it is possible to send messages larger than max_notification_size of the Validation protocol.

In unify_with_peer, we send all the assignments we've seen for all candidates at a given block which just came into the view of a peer, and with aggresion mode active this is even a bigger problem.

A wire message looks like this:

pub enum ApprovalDistributionMessage {
		/// Assignments for candidates in recent, unfinalized blocks.
		///
		/// Actually checking the assignment may yield a different result.
		#[codec(index = 0)]
		Assignments(Vec<(IndirectAssignmentCert, CandidateIndex)>),
		/// Approvals for candidates in some recent, unfinalized block.
		#[codec(index = 1)]
		Approvals(Vec<IndirectSignedApprovalVote>),
	}

If a peer is behind with 1 block, we're gonna send all received assignments and approvals for all candidates up to 12th tranche.

In the case of assignments, which are larger, we can observe a wire size of 132 bytes plus encoding overhead:
32 bytes VRF output, 64 bytes VRF proof, 32 bytes block hash, 4 bytes validator index, 4 byte sample or core index
If we assume 50 paras and first tranches covering needed_approvals (30) , that is 198.000 bytes > 100KB max notification size
Adding in more validators given tranche distribution staying the same, we would get even larger messages.

We need to break these huge messages down in batches to something lower that what the protocol allows.

[ordian]

needed approvals tranches (30)

needed_approvals is measured in approvals, not tranches. Typically, we'd need the first couple of tranches to cover them (most in 0th tranche).

[sandreim]

needed approvals tranches (30)

needed_approvals is measured in approvals, not tranches. Typically, we'd need the first couple of tranches to cover them (most in 0th tranche).

corrected, that was what I was aiming at, poor choice of words

[burdges]

This is forwarding batches from multiple authorities between nodes, right?

[ordian]

The title is slightly misleading. What is does is opposite of batching, namely, it splits the assignment packets into chunks so they always fit into substrate/libp2p max_notification_size limit.

[burdges]

Ahh okay, but an assignment packet should be like 200 bytes, no? public key (32) + VRF signature (96) + candidate (32) + a few bytes describing the sampling. Or you need to attach the candidate receipt to all of them?

[ordian]

No, we do no attach the receipt, but

In unify_with_peer, we send all the assignments we've seen for all candidates at a given block which just came into the view of a peer, and with aggresion mode active this is even a bigger problem.

So a single message can contain multiple assignments.

Actually, for peer view changes, maybe we shouldn't propagate all assignments if this is our grid neighbor, it seems quite wasteful. But this is getting off-topic.