Improve connection distribution in clusters to reduce latency and reactor utilization

[ballard26]

During testing with OMB on a 3x i3en.6xlarge cluster it was found that traffic from internal RPC clients can cause reactor utilization on some shards to be 40% higher than others. This largely caused by poor distribution of client connections to other brokers amongst shards.

Looking at RPC client in + out bytes in the graph below we can see that some shards are processing 4x the amount of throughput as others which can explain a lot of the extra reactor utilization on some shards.

In the chart below the allowed connections was set to equal the number of shards on a given broker. This allows each shard to have its own connections to all of the brokers in the cluster. And we can see in the chart that the standard deviation for reactor utilization is reduced by 50%.

[mattschumpert]

@StephanDollberg can we update the title of this issue to represent the improvement the PR(s) will make please?

[piyushredpanda]

That's on @ballard26 to do. This will help with reducing variance in latencies.

[ballard26]

I've updated the title let me know if it's adequate.

[mattschumpert]

💥