fix(security): Randomly drop connections when inbound service is overloaded

[teor2345]

Motivation

Zebra should avoid dropping all peer connections when the inbound service is overloaded by a subset of peers.

This PR will randomly drop connections that get an overloaded error, with a high probability

Closes #6596.

Solution

• Randomly drop connections that receive overloaded errors, with greater likelihood when a second overloaded error is seen soon after the first.
• Add a ServiceShutdown variant to PeerError for when the inbound service has failed or shutdown.

Testing:

• more overloads mean a higher probability of being dropped
• less overloads mean a lower probability
• frequent overloads significantly increase the probability
• when we make multiple connections, some are dropped and some aren't

Related fixes:

• add some test utility methods and fix docs

Review

This is a routine fix.

The production code has already been reviewed privately, the tests are new.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

• Use a skiplist to keep an ordered buffer queue

[codecov]

Codecov Report

Merging #6790 (ca23d02) into main (6f8c981) will increase coverage by 0.17%.
The diff coverage is 78.68%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6790      +/-   ##
==========================================
+ Coverage   77.99%   78.17%   +0.17%     
==========================================
  Files         308      308              
  Lines       40975    41023      +48     
==========================================
+ Hits        31960    32071     +111     
+ Misses       9015     8952      -63     

[teor2345]

Sorry for the doctest compile errors, I didn't catch them locally.

(Next time I'll fix the test API separately.)