Prevent DoS attacks based on Extensible payloads

[roman-khimov]

Summary or problem description
Extensible P2P payloads introduced in preview5 can be sent by a number of whitelisted senders, currently including:

• consensus nodes
• oracle nodes
• state validators

Any of these nodes can send an extensible payload and it'll be broadcasted through the network. It can send 2/3/many of them and there is nothing limiting particular node in doing that. Before preview5 we had similar issue with consensus payloads, but they were limited to consensus nodes, a small list of addresses that arguably are the most trusted addresses on the network. Now we have more potential senders, so there is a higher risk of some of them flooding the network with Extensible payloads either because of some bug or for malicious purposes.

All current Extensible payloads are stored in node's RelayCache which operates as FIFO cache with capacity for 100 items, so it's really easily to kick out any other payload with just a 100 of messages and keep the network busy with Extensible broadcasts. It's relatively easy to detect this behavior, but to remove some address from whitelist we need a committee-signed transaction reassigning some role (or voting transaction for CNs). But even after the appropriate transaction is constructed it might be seriously delayed because RelayCache also stores consensus messages that can be kicked out of it by flooder preventing proper consensus message propagation. So Extensible payload misuse can stop the network for some time.

Do you have any solution you want to propose?

1. Limit the number of nodes that can use Extensible payloads.
   Oracles don't use Extensible payloads at the moment and yet they're in the list, removing them might help a bit. As well as not adding new roles there unless they use Extensible payloads for some purpose.
2. Limit the number of messages stored in RelayCache for one sender address.
   Like one sender can't occupy more than 10% of this pool.

Of course some staking/deposit mechanism could also be used instead, but maybe it's not yet needed for these roles.

Neo Version

• Neo 3

Where in the software does this update applies to?

• P2P (TCP)